Note: This is test shard 8 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from TableRangeRebalancingTest
[ RUN ] TableRangeRebalancingTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 14:11:55.878859 10820 test_util.cc:268] Using random seed: 1297787288
I20240503 14:11:55.891817 10820 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:11:55.891918 10820 ts_itest-base.cc:113] --------------
I20240503 14:11:55.891957 10820 ts_itest-base.cc:114] 3 tablet servers
I20240503 14:11:55.892040 10820 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:11:55.892098 10820 ts_itest-base.cc:116] --------------
2024-05-03T14:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:11:55Z Disabled control of system clock
I20240503 14:11:55.916687 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:40867
--webserver_interface=127.10.145.62
--webserver_port=0
--builtin_ntp_servers=127.10.145.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.145.62:40867
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 14:11:56.278402 10832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.278735 10832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.278846 10832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.287016 10832 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:56.287164 10832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:11:56.287246 10832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.287310 10832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:11:56.287354 10832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:11:56.297400 10832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.10.145.62:40867
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.298846 10832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.300865 10832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.312456 10838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.312605 10837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.312811 10840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.313797 10832 server_base.cc:1034] running on GCE node
I20240503 14:11:56.314759 10832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.316685 10832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.317940 10832 hybrid_clock.cc:648] HybridClock initialized: now 1714745516317884 us; error 68 us; skew 500 ppm
I20240503 14:11:56.318361 10832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.325224 10832 webserver.cc:460] Webserver started at http://127.10.145.62:36891/ using document root <none> and password file <none>
I20240503 14:11:56.326123 10832 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.326249 10832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.326539 10832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.329483 10832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9fe84d6d8aa04795b288795572da72b6"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-v549"
I20240503 14:11:56.330375 10832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9fe84d6d8aa04795b288795572da72b6"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-v549"
I20240503 14:11:56.336716 10832 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240503 14:11:56.340719 10846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.342427 10832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:56.342619 10832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9fe84d6d8aa04795b288795572da72b6"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-v549"
I20240503 14:11:56.342823 10832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.364058 10832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.364992 10832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.365341 10832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.386055 10832 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.62:40867
I20240503 14:11:56.386088 10898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.62:40867 every 8 connection(s)
I20240503 14:11:56.387647 10832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:11:56.392055 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:56.395491 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 10832
I20240503 14:11:56.395817 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:11:56.405403 10899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6: Bootstrap starting.
I20240503 14:11:56.408938 10899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:56.410252 10899 log.cc:824] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:56.413520 10899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6: No bootstrap required, opened a new log
I20240503 14:11:56.422366 10899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe84d6d8aa04795b288795572da72b6" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 40867 } }
I20240503 14:11:56.422960 10899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:56.423640 10899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe84d6d8aa04795b288795572da72b6, State: Initialized, Role: FOLLOWER
I20240503 14:11:56.424549 10899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe84d6d8aa04795b288795572da72b6" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 40867 } }
I20240503 14:11:56.424860 10899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:11:56.425031 10899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:11:56.425256 10899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:56.428437 10899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe84d6d8aa04795b288795572da72b6" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 40867 } }
I20240503 14:11:56.429047 10899 leader_election.cc:302] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fe84d6d8aa04795b288795572da72b6; no voters:
I20240503 14:11:56.429739 10899 leader_election.cc:288] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:11:56.429913 10904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:56.430997 10904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [term 1 LEADER]: Becoming Leader. State: Replica: 9fe84d6d8aa04795b288795572da72b6, State: Running, Role: LEADER
I20240503 14:11:56.431660 10904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe84d6d8aa04795b288795572da72b6" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 40867 } }
I20240503 14:11:56.432227 10899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:56.434957 10905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fe84d6d8aa04795b288795572da72b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe84d6d8aa04795b288795572da72b6" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 40867 } } }
I20240503 14:11:56.435369 10905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:56.436640 10906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fe84d6d8aa04795b288795572da72b6. Latest consensus state: current_term: 1 leader_uuid: "9fe84d6d8aa04795b288795572da72b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe84d6d8aa04795b288795572da72b6" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 40867 } } }
I20240503 14:11:56.436996 10906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:56.439952 10909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:11:56.446563 10909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:11:56.457463 10909 catalog_manager.cc:1347] Generated new cluster ID: 75ce4aba78dd421f9faeaa877df97836
I20240503 14:11:56.457607 10909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:11:56.476091 10909 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:11:56.478008 10909 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:11:56.502749 10909 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9fe84d6d8aa04795b288795572da72b6: Generated new TSK 0
I20240503 14:11:56.503825 10909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:11:56.517854 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.1:0
--local_ip_for_outbound_sockets=127.10.145.1
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:40867
--builtin_ntp_servers=127.10.145.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:11:56.890439 10923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.890770 10923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.890904 10923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.899313 10923 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:56.899454 10923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.899618 10923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.1
I20240503 14:11:56.909376 10923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:40867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.1
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.910795 10923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.913201 10923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.925623 10931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.925724 10928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.925746 10929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.926225 10923 server_base.cc:1034] running on GCE node
I20240503 14:11:56.927537 10923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.928560 10923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.929764 10923 hybrid_clock.cc:648] HybridClock initialized: now 1714745516929714 us; error 48 us; skew 500 ppm
I20240503 14:11:56.930099 10923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.940186 10923 webserver.cc:460] Webserver started at http://127.10.145.1:36691/ using document root <none> and password file <none>
I20240503 14:11:56.941569 10923 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.941793 10923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.942262 10923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.946223 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "85aecf73844f45aab998a795718f6824"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-v549"
I20240503 14:11:56.947283 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "85aecf73844f45aab998a795718f6824"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-v549"
I20240503 14:11:56.955879 10923 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20240503 14:11:56.961171 10937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.963222 10923 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:11:56.963438 10923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "85aecf73844f45aab998a795718f6824"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-v549"
I20240503 14:11:56.963693 10923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.988708 10923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.989719 10923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.989984 10923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.990967 10923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:56.992534 10923 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:56.992625 10923 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.993208 10923 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:56.993264 10923 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.027302 10923 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.1:32899
I20240503 14:11:57.027423 11050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.1:32899 every 8 connection(s)
I20240503 14:11:57.028743 10923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:11:57.031912 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 10923
I20240503 14:11:57.032140 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:11:57.035705 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.2:0
--local_ip_for_outbound_sockets=127.10.145.2
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:40867
--builtin_ntp_servers=127.10.145.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:11:57.044097 11051 heartbeater.cc:347] Connected to a master server at 127.10.145.62:40867
I20240503 14:11:57.044445 11051 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:57.045403 11051 heartbeater.cc:511] Master 127.10.145.62:40867 requested a full tablet report, sending...
I20240503 14:11:57.048019 10864 ts_manager.cc:194] Registered new tserver with Master: 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899)
I20240503 14:11:57.050392 10864 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.1:53997
W20240503 14:11:57.392319 11055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.392619 11055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.392735 11055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.400925 11055 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:57.401062 11055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.401227 11055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.2
I20240503 14:11:57.411072 11055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:40867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.2
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.412503 11055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.414402 11055 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.428458 11061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.428539 11060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.428560 11063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.429697 11055 server_base.cc:1034] running on GCE node
I20240503 14:11:57.430387 11055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.431432 11055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.432646 11055 hybrid_clock.cc:648] HybridClock initialized: now 1714745517432580 us; error 66 us; skew 500 ppm
I20240503 14:11:57.433007 11055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.435529 11055 webserver.cc:460] Webserver started at http://127.10.145.2:38129/ using document root <none> and password file <none>
I20240503 14:11:57.436334 11055 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.436452 11055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.436772 11055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.439661 11055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-v549"
I20240503 14:11:57.440537 11055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-v549"
I20240503 14:11:57.446202 11055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240503 14:11:57.450376 11069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.451998 11055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20240503 14:11:57.452203 11055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-v549"
I20240503 14:11:57.452421 11055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:57.477909 11055 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:57.478863 11055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:57.479182 11055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:57.480242 11055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:57.481974 11055 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:57.482084 11055 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.482209 11055 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:57.482277 11055 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.517752 11055 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.2:36853
I20240503 14:11:57.518302 11182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.2:36853 every 8 connection(s)
I20240503 14:11:57.519199 11055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:11:57.524284 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 11055
I20240503 14:11:57.524441 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:11:57.527830 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.3:0
--local_ip_for_outbound_sockets=127.10.145.3
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:40867
--builtin_ntp_servers=127.10.145.20:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:11:57.533661 11183 heartbeater.cc:347] Connected to a master server at 127.10.145.62:40867
I20240503 14:11:57.534024 11183 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:57.534986 11183 heartbeater.cc:511] Master 127.10.145.62:40867 requested a full tablet report, sending...
I20240503 14:11:57.536873 10864 ts_manager.cc:194] Registered new tserver with Master: bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:11:57.538093 10864 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.2:39433
W20240503 14:11:57.896323 11186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.896652 11186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.896749 11186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.905148 11186 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:57.905301 11186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.905443 11186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.3
I20240503 14:11:57.915156 11186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:40867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.3
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.916534 11186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.918540 11186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.933146 11194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.933256 11191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.933230 11192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.934033 11186 server_base.cc:1034] running on GCE node
I20240503 14:11:57.934777 11186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.935869 11186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.937117 11186 hybrid_clock.cc:648] HybridClock initialized: now 1714745517937055 us; error 58 us; skew 500 ppm
I20240503 14:11:57.937539 11186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.940129 11186 webserver.cc:460] Webserver started at http://127.10.145.3:39993/ using document root <none> and password file <none>
I20240503 14:11:57.941049 11186 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.941210 11186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.941582 11186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.944447 11186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1eaac79055b14e99b0f6d1097cae21ce"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-v549"
I20240503 14:11:57.945379 11186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1eaac79055b14e99b0f6d1097cae21ce"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-v549"
I20240503 14:11:57.951335 11186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20240503 14:11:57.955487 11200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.957118 11186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:11:57.957374 11186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1eaac79055b14e99b0f6d1097cae21ce"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-v549"
I20240503 14:11:57.957605 11186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:57.982837 11186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:57.983773 11186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:57.984076 11186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:57.985126 11186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:57.986891 11186 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:57.986999 11186 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.987114 11186 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:57.987190 11186 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.022137 11186 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.3:33607
I20240503 14:11:58.022228 11313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.3:33607 every 8 connection(s)
I20240503 14:11:58.023553 11186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:11:58.027412 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 11186
I20240503 14:11:58.027611 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:11:58.039383 11314 heartbeater.cc:347] Connected to a master server at 127.10.145.62:40867
I20240503 14:11:58.039764 11314 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.041206 11314 heartbeater.cc:511] Master 127.10.145.62:40867 requested a full tablet report, sending...
I20240503 14:11:58.043058 10864 ts_manager.cc:194] Registered new tserver with Master: 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:11:58.044211 10864 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.3:40135
I20240503 14:11:58.045738 10820 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 14:11:58.053958 11051 heartbeater.cc:503] Master 127.10.145.62:40867 was elected leader, sending a full tablet report...
I20240503 14:11:58.069067 10820 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 14:11:58.541534 11183 heartbeater.cc:503] Master 127.10.145.62:40867 was elected leader, sending a full tablet report...
I20240503 14:11:59.047658 11314 heartbeater.cc:503] Master 127.10.145.62:40867 was elected leader, sending a full tablet report...
I20240503 14:12:01.823742 10864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47654:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\200\007\001\000\000\000\000\000\006\001\000\000\000\000\000\007\001\000\377\377\377\177"
indirect_data: "<redacted>"""
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 8
seed: 0
}
}
W20240503 14:12:01.831068 10864 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:12:01.879490 11106 tablet_service.cc:1467] Processing CreateTablet for tablet 28e12492b6da43abab399c40c0232c9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.879933 11108 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6ce01ac44f4b4a8aac787f03d83af3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.880592 11112 tablet_service.cc:1467] Processing CreateTablet for tablet e834edfd67b54b6fbc62c3c848100708 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.880867 11106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e12492b6da43abab399c40c0232c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.881052 11114 tablet_service.cc:1467] Processing CreateTablet for tablet b7730fcf05b144afb560055fb4802e57 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.881580 11112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e834edfd67b54b6fbc62c3c848100708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.881533 11247 tablet_service.cc:1467] Processing CreateTablet for tablet 204b25d73f784c2385cd2996d7fdaa5b (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.882707 11246 tablet_service.cc:1467] Processing CreateTablet for tablet b7730fcf05b144afb560055fb4802e57 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.883592 11246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7730fcf05b144afb560055fb4802e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.884119 11249 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5c86d427064ebc803628e9269f4fe0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.884678 11249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5c86d427064ebc803628e9269f4fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.885110 11114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7730fcf05b144afb560055fb4802e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.885353 11248 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac3f87f180472680a7f09b0ceef0b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.886070 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ac3f87f180472680a7f09b0ceef0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.886070 11108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6ce01ac44f4b4a8aac787f03d83af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.885879 11111 tablet_service.cc:1467] Processing CreateTablet for tablet fc770ed0b57a4a579abb38417866f3fb (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.878463 11110 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9ed0f4fe8f4218b88e0ebec5034ae3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.887225 11107 tablet_service.cc:1467] Processing CreateTablet for tablet d90da99a633945fcaf6a1740d64e6d50 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.887244 11244 tablet_service.cc:1467] Processing CreateTablet for tablet e834edfd67b54b6fbc62c3c848100708 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.888062 11244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e834edfd67b54b6fbc62c3c848100708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.888125 11243 tablet_service.cc:1467] Processing CreateTablet for tablet fc770ed0b57a4a579abb38417866f3fb (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.889268 11115 tablet_service.cc:1467] Processing CreateTablet for tablet 204b25d73f784c2385cd2996d7fdaa5b (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.889657 11107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90da99a633945fcaf6a1740d64e6d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.890875 11116 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac3f87f180472680a7f09b0ceef0b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.890918 11104 tablet_service.cc:1467] Processing CreateTablet for tablet 554b725e046a4dbbae9b4a6126c95389 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.889309 11105 tablet_service.cc:1467] Processing CreateTablet for tablet 3e92a0269bf8444bb814daba5ea0c701 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.891492 11116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ac3f87f180472680a7f09b0ceef0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.892277 11103 tablet_service.cc:1467] Processing CreateTablet for tablet c97ff554385741ada59c1d3779ef008f (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.888428 11109 tablet_service.cc:1467] Processing CreateTablet for tablet 89c7203d4650496188b9d84858bfcc35 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.888862 11113 tablet_service.cc:1467] Processing CreateTablet for tablet 06166dc9654d408193d9734634654673 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.892897 11115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204b25d73f784c2385cd2996d7fdaa5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.893118 11102 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3a8130076746a88b31f2bd3ceab7c9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.893882 11243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc770ed0b57a4a579abb38417866f3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.882058 10976 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6ce01ac44f4b4a8aac787f03d83af3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.886708 10977 tablet_service.cc:1467] Processing CreateTablet for tablet 89c7203d4650496188b9d84858bfcc35 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.884102 10975 tablet_service.cc:1467] Processing CreateTablet for tablet d90da99a633945fcaf6a1740d64e6d50 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.892290 10971 tablet_service.cc:1467] Processing CreateTablet for tablet c97ff554385741ada59c1d3779ef008f (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.894933 10970 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3a8130076746a88b31f2bd3ceab7c9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.883603 10979 tablet_service.cc:1467] Processing CreateTablet for tablet fc770ed0b57a4a579abb38417866f3fb (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.890316 10972 tablet_service.cc:1467] Processing CreateTablet for tablet 554b725e046a4dbbae9b4a6126c95389 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.883098 10981 tablet_service.cc:1467] Processing CreateTablet for tablet 06166dc9654d408193d9734634654673 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.889803 10974 tablet_service.cc:1467] Processing CreateTablet for tablet 28e12492b6da43abab399c40c0232c9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.880573 10982 tablet_service.cc:1467] Processing CreateTablet for tablet b7730fcf05b144afb560055fb4802e57 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.881901 10985 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5c86d427064ebc803628e9269f4fe0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.890552 10980 tablet_service.cc:1467] Processing CreateTablet for tablet e834edfd67b54b6fbc62c3c848100708 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.882593 10983 tablet_service.cc:1467] Processing CreateTablet for tablet 204b25d73f784c2385cd2996d7fdaa5b (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.890239 10973 tablet_service.cc:1467] Processing CreateTablet for tablet 3e92a0269bf8444bb814daba5ea0c701 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.887928 10978 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9ed0f4fe8f4218b88e0ebec5034ae3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.893537 10984 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac3f87f180472680a7f09b0ceef0b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.896239 10971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97ff554385741ada59c1d3779ef008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.896890 10984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ac3f87f180472680a7f09b0ceef0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.897303 10970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c3a8130076746a88b31f2bd3ceab7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.897758 10979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc770ed0b57a4a579abb38417866f3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.898204 11105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e92a0269bf8444bb814daba5ea0c701. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.899258 10972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 554b725e046a4dbbae9b4a6126c95389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.899544 10973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e92a0269bf8444bb814daba5ea0c701. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.899830 10983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204b25d73f784c2385cd2996d7fdaa5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.900055 10981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06166dc9654d408193d9734634654673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.900326 10978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9ed0f4fe8f4218b88e0ebec5034ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.900272 11117 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5c86d427064ebc803628e9269f4fe0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.900538 10977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c7203d4650496188b9d84858bfcc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.901928 10976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6ce01ac44f4b4a8aac787f03d83af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.902261 10974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e12492b6da43abab399c40c0232c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.902532 10975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90da99a633945fcaf6a1740d64e6d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.903239 10980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e834edfd67b54b6fbc62c3c848100708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.903952 10985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5c86d427064ebc803628e9269f4fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.910264 11113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06166dc9654d408193d9734634654673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.911257 11103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97ff554385741ada59c1d3779ef008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.912236 11102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c3a8130076746a88b31f2bd3ceab7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.913273 11109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c7203d4650496188b9d84858bfcc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.913898 10982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7730fcf05b144afb560055fb4802e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.914124 11110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9ed0f4fe8f4218b88e0ebec5034ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.914919 11104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 554b725e046a4dbbae9b4a6126c95389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.891439 11241 tablet_service.cc:1467] Processing CreateTablet for tablet 89c7203d4650496188b9d84858bfcc35 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.893775 11242 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9ed0f4fe8f4218b88e0ebec5034ae3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.916960 11111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc770ed0b57a4a579abb38417866f3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.925908 11117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5c86d427064ebc803628e9269f4fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.919385 11239 tablet_service.cc:1467] Processing CreateTablet for tablet d90da99a633945fcaf6a1740d64e6d50 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.915650 11245 tablet_service.cc:1467] Processing CreateTablet for tablet 06166dc9654d408193d9734634654673 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.919564 11236 tablet_service.cc:1467] Processing CreateTablet for tablet 554b725e046a4dbbae9b4a6126c95389 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.919906 11234 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3a8130076746a88b31f2bd3ceab7c9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.932523 11241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89c7203d4650496188b9d84858bfcc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.919687 11238 tablet_service.cc:1467] Processing CreateTablet for tablet 28e12492b6da43abab399c40c0232c9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.919791 11235 tablet_service.cc:1467] Processing CreateTablet for tablet c97ff554385741ada59c1d3779ef008f (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.935200 11239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90da99a633945fcaf6a1740d64e6d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.920989 11240 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6ce01ac44f4b4a8aac787f03d83af3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 14:12:01.934332 11237 tablet_service.cc:1467] Processing CreateTablet for tablet 3e92a0269bf8444bb814daba5ea0c701 (DEFAULT_TABLE table=rebalance_test_table_0 [id=de8e8529556b4356a514cac11c756833]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 14:12:01.941607 11247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204b25d73f784c2385cd2996d7fdaa5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.949600 11238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e12492b6da43abab399c40c0232c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.957372 11236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 554b725e046a4dbbae9b4a6126c95389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.959949 11245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06166dc9654d408193d9734634654673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.965409 11237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e92a0269bf8444bb814daba5ea0c701. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.966522 11235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97ff554385741ada59c1d3779ef008f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.968825 11234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c3a8130076746a88b31f2bd3ceab7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.969839 11242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9ed0f4fe8f4218b88e0ebec5034ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.970784 11240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6ce01ac44f4b4a8aac787f03d83af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.014833 11332 tablet_bootstrap.cc:492] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.016306 11334 tablet_bootstrap.cc:492] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.019335 11332 tablet_bootstrap.cc:654] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.020545 11334 tablet_bootstrap.cc:654] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.021653 11332 log.cc:824] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:02.023125 11334 log.cc:824] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:02.070761 11333 tablet_bootstrap.cc:492] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.073318 11334 tablet_bootstrap.cc:492] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.073885 11334 ts_tablet_manager.cc:1367] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.058s user 0.022s sys 0.010s
I20240503 14:12:02.074025 11332 tablet_bootstrap.cc:492] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.074666 11332 ts_tablet_manager.cc:1367] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.060s user 0.018s sys 0.040s
I20240503 14:12:02.075222 11333 tablet_bootstrap.cc:654] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.076707 11333 log.cc:824] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:02.080451 11333 tablet_bootstrap.cc:492] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.080796 11333 ts_tablet_manager.cc:1367] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 14:12:02.085752 11334 raft_consensus.cc:348] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.086863 11334 raft_consensus.cc:374] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.087194 11334 raft_consensus.cc:729] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.088310 11333 raft_consensus.cc:348] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.088809 11333 raft_consensus.cc:374] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.088619 11334 consensus_queue.cc:260] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.088925 11333 raft_consensus.cc:729] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.088912 11332 raft_consensus.cc:348] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.089953 11332 raft_consensus.cc:374] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.089910 11333 consensus_queue.cc:260] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.090111 11332 raft_consensus.cc:729] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.090976 11334 ts_tablet_manager.cc:1398] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20240503 14:12:02.091063 11332 consensus_queue.cc:260] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.092797 11333 ts_tablet_manager.cc:1398] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20240503 14:12:02.093329 11333 tablet_bootstrap.cc:492] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.094549 11334 tablet_bootstrap.cc:492] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.096459 11333 tablet_bootstrap.cc:654] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.096618 11334 tablet_bootstrap.cc:654] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.099481 11332 ts_tablet_manager.cc:1398] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.025s user 0.016s sys 0.005s
I20240503 14:12:02.100121 11332 tablet_bootstrap.cc:492] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.101189 11334 tablet_bootstrap.cc:492] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.101409 11334 ts_tablet_manager.cc:1367] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:12:02.102134 11332 tablet_bootstrap.cc:654] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.103049 11333 tablet_bootstrap.cc:492] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.102999 11334 raft_consensus.cc:348] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.103245 11334 raft_consensus.cc:374] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.103348 11334 raft_consensus.cc:729] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.103502 11333 ts_tablet_manager.cc:1367] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240503 14:12:02.103590 11334 consensus_queue.cc:260] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.104738 11334 ts_tablet_manager.cc:1398] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 14:12:02.106546 11333 raft_consensus.cc:348] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.106902 11333 raft_consensus.cc:374] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.107035 11333 raft_consensus.cc:729] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.107342 11333 consensus_queue.cc:260] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.107724 11334 tablet_bootstrap.cc:492] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.108182 11333 ts_tablet_manager.cc:1398] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:02.108582 11333 tablet_bootstrap.cc:492] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.109833 11334 tablet_bootstrap.cc:654] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.111214 11332 tablet_bootstrap.cc:492] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.111450 11332 ts_tablet_manager.cc:1367] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240503 14:12:02.112912 11333 tablet_bootstrap.cc:654] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.112886 11332 raft_consensus.cc:348] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.113147 11332 raft_consensus.cc:374] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.113241 11332 raft_consensus.cc:729] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.113510 11332 consensus_queue.cc:260] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.114069 11332 ts_tablet_manager.cc:1398] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 14:12:02.114400 11332 tablet_bootstrap.cc:492] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.116258 11332 tablet_bootstrap.cc:654] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.121938 11332 tablet_bootstrap.cc:492] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.122123 11332 ts_tablet_manager.cc:1367] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240503 14:12:02.122419 11334 tablet_bootstrap.cc:492] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.122434 11333 tablet_bootstrap.cc:492] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.122607 11334 ts_tablet_manager.cc:1367] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20240503 14:12:02.122608 11333 ts_tablet_manager.cc:1367] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20240503 14:12:02.122856 11332 raft_consensus.cc:348] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.123064 11332 raft_consensus.cc:374] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.123138 11332 raft_consensus.cc:729] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.123385 11332 consensus_queue.cc:260] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.123340 11334 raft_consensus.cc:348] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.123342 11333 raft_consensus.cc:348] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } }
I20240503 14:12:02.123566 11333 raft_consensus.cc:374] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.123566 11334 raft_consensus.cc:374] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.123647 11334 raft_consensus.cc:729] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.123675 11333 raft_consensus.cc:729] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.123891 11334 consensus_queue.cc:260] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.123898 11333 consensus_queue.cc:260] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } }
I20240503 14:12:02.124022 11332 ts_tablet_manager.cc:1398] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:02.124305 11332 tablet_bootstrap.cc:492] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.124434 11333 ts_tablet_manager.cc:1398] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:02.124431 11334 ts_tablet_manager.cc:1398] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:02.124715 11334 tablet_bootstrap.cc:492] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.124745 11333 tablet_bootstrap.cc:492] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.125958 11332 tablet_bootstrap.cc:654] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.126325 11333 tablet_bootstrap.cc:654] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.126325 11334 tablet_bootstrap.cc:654] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.134807 11334 tablet_bootstrap.cc:492] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.134976 11334 ts_tablet_manager.cc:1367] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240503 14:12:02.135193 11332 tablet_bootstrap.cc:492] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.135342 11332 ts_tablet_manager.cc:1367] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20240503 14:12:02.135753 11333 tablet_bootstrap.cc:492] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.135783 11334 raft_consensus.cc:348] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.135915 11333 ts_tablet_manager.cc:1367] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 14:12:02.135998 11334 raft_consensus.cc:374] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.135958 11332 raft_consensus.cc:348] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.136097 11334 raft_consensus.cc:729] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.136159 11332 raft_consensus.cc:374] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.136247 11332 raft_consensus.cc:729] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.136348 11334 consensus_queue.cc:260] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.136461 11332 consensus_queue.cc:260] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.137009 11332 ts_tablet_manager.cc:1398] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:02.137029 11334 ts_tablet_manager.cc:1398] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:12:02.137313 11332 tablet_bootstrap.cc:492] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.137339 11334 tablet_bootstrap.cc:492] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.137264 11333 raft_consensus.cc:348] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.137498 11333 raft_consensus.cc:374] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.137579 11333 raft_consensus.cc:729] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.137812 11333 consensus_queue.cc:260] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.139132 11334 tablet_bootstrap.cc:654] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.139496 11333 ts_tablet_manager.cc:1398] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:02.139551 11332 tablet_bootstrap.cc:654] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.139784 11333 tablet_bootstrap.cc:492] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.141737 11333 tablet_bootstrap.cc:654] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.143214 11338 raft_consensus.cc:482] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.143471 11338 raft_consensus.cc:504] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.145695 11338 leader_election.cc:288] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.148588 11332 tablet_bootstrap.cc:492] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.148818 11332 ts_tablet_manager.cc:1367] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240503 14:12:02.149663 11332 raft_consensus.cc:348] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.149914 11332 raft_consensus.cc:374] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.149989 11332 raft_consensus.cc:729] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.150303 11332 consensus_queue.cc:260] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.150975 11332 ts_tablet_manager.cc:1398] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:02.151305 11332 tablet_bootstrap.cc:492] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.152359 11334 tablet_bootstrap.cc:492] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.152534 11334 ts_tablet_manager.cc:1367] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240503 14:12:02.153136 11332 tablet_bootstrap.cc:654] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.153369 11334 raft_consensus.cc:348] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.153508 11333 tablet_bootstrap.cc:492] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.153595 11334 raft_consensus.cc:374] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.153663 11334 raft_consensus.cc:729] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.153672 11333 ts_tablet_manager.cc:1367] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20240503 14:12:02.153914 11334 consensus_queue.cc:260] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.154436 11333 raft_consensus.cc:348] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.154551 11334 ts_tablet_manager.cc:1398] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:02.154662 11333 raft_consensus.cc:374] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.154749 11333 raft_consensus.cc:729] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.154865 11334 tablet_bootstrap.cc:492] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.155022 11333 consensus_queue.cc:260] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.158092 11334 tablet_bootstrap.cc:654] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.159122 11333 ts_tablet_manager.cc:1398] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:02.159601 11333 tablet_bootstrap.cc:492] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.159678 11332 tablet_bootstrap.cc:492] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.159821 11332 ts_tablet_manager.cc:1367] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 14:12:02.161381 11332 raft_consensus.cc:348] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.161618 11332 raft_consensus.cc:374] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.161708 11332 raft_consensus.cc:729] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.161979 11332 consensus_queue.cc:260] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.162606 11333 tablet_bootstrap.cc:654] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.162680 11332 ts_tablet_manager.cc:1398] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:02.162966 11332 tablet_bootstrap.cc:492] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.164710 11332 tablet_bootstrap.cc:654] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.167809 11333 tablet_bootstrap.cc:492] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.167878 11334 tablet_bootstrap.cc:492] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.167984 11333 ts_tablet_manager.cc:1367] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 14:12:02.168022 11334 ts_tablet_manager.cc:1367] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240503 14:12:02.168792 11334 raft_consensus.cc:348] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.168794 11333 raft_consensus.cc:348] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.169008 11333 raft_consensus.cc:374] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.169026 11334 raft_consensus.cc:374] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.169093 11333 raft_consensus.cc:729] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.169126 11334 raft_consensus.cc:729] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.169402 11333 consensus_queue.cc:260] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.169447 11334 consensus_queue.cc:260] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.170778 11334 ts_tablet_manager.cc:1398] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:02.170953 11333 ts_tablet_manager.cc:1398] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:02.171089 11334 tablet_bootstrap.cc:492] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.171228 11333 tablet_bootstrap.cc:492] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.171463 11332 tablet_bootstrap.cc:492] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.171671 11332 ts_tablet_manager.cc:1367] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240503 14:12:02.172534 11332 raft_consensus.cc:348] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } }
I20240503 14:12:02.172780 11332 raft_consensus.cc:374] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.172850 11332 raft_consensus.cc:729] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.172895 11334 tablet_bootstrap.cc:654] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.173096 11332 consensus_queue.cc:260] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } }
I20240503 14:12:02.173835 11332 ts_tablet_manager.cc:1398] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:02.174152 11332 tablet_bootstrap.cc:492] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.174217 11333 tablet_bootstrap.cc:654] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.176095 11332 tablet_bootstrap.cc:654] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.178438 11332 tablet_bootstrap.cc:492] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.178607 11332 ts_tablet_manager.cc:1367] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 14:12:02.179477 11332 raft_consensus.cc:348] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.179716 11332 raft_consensus.cc:374] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.179807 11332 raft_consensus.cc:729] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.180086 11332 consensus_queue.cc:260] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.180749 11332 ts_tablet_manager.cc:1398] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:02.181047 11332 tablet_bootstrap.cc:492] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.182853 11332 tablet_bootstrap.cc:654] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.185256 11334 tablet_bootstrap.cc:492] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.185482 11334 ts_tablet_manager.cc:1367] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.002s
I20240503 14:12:02.185730 11333 tablet_bootstrap.cc:492] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.185918 11333 ts_tablet_manager.cc:1367] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20240503 14:12:02.186823 11334 raft_consensus.cc:348] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.187120 11334 raft_consensus.cc:374] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.187225 11334 raft_consensus.cc:729] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.187304 11333 raft_consensus.cc:348] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.187593 11333 raft_consensus.cc:374] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.187498 11334 consensus_queue.cc:260] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.187685 11333 raft_consensus.cc:729] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.187693 11005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90da99a633945fcaf6a1740d64e6d50" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.188134 11334 ts_tablet_manager.cc:1398] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:02.188015 11333 consensus_queue.cc:260] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.188463 11334 tablet_bootstrap.cc:492] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.188653 11333 ts_tablet_manager.cc:1398] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 14:12:02.189025 11333 tablet_bootstrap.cc:492] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
W20240503 14:12:02.190433 11072 leader_election.cc:341] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.190918 11333 tablet_bootstrap.cc:654] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.192338 11334 tablet_bootstrap.cc:654] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.192867 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90da99a633945fcaf6a1740d64e6d50" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
W20240503 14:12:02.195039 11070 leader_election.cc:341] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.195240 11070 leader_election.cc:302] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfaac921af9f4b5dbc23321fbfb4d09e; no voters: 1eaac79055b14e99b0f6d1097cae21ce, 85aecf73844f45aab998a795718f6824
I20240503 14:12:02.195827 11332 tablet_bootstrap.cc:492] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.196022 11332 ts_tablet_manager.cc:1367] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20240503 14:12:02.196223 11338 raft_consensus.cc:2726] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.196760 11332 raft_consensus.cc:348] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.197027 11332 raft_consensus.cc:374] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.197110 11332 raft_consensus.cc:729] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.197448 11333 tablet_bootstrap.cc:492] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.197429 11332 consensus_queue.cc:260] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.197602 11333 ts_tablet_manager.cc:1367] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20240503 14:12:02.198120 11334 tablet_bootstrap.cc:492] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.198258 11332 ts_tablet_manager.cc:1398] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:02.198280 11334 ts_tablet_manager.cc:1367] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20240503 14:12:02.198434 11333 raft_consensus.cc:348] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.198622 11332 tablet_bootstrap.cc:492] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.198642 11333 raft_consensus.cc:374] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.198721 11333 raft_consensus.cc:729] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.198993 11333 consensus_queue.cc:260] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.199275 11334 raft_consensus.cc:348] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.199467 11334 raft_consensus.cc:374] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.199463 11340 raft_consensus.cc:482] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.199551 11334 raft_consensus.cc:729] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.199710 11333 ts_tablet_manager.cc:1398] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:02.199707 11340 raft_consensus.cc:504] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.199759 11334 consensus_queue.cc:260] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.199999 11333 tablet_bootstrap.cc:492] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.200978 11334 ts_tablet_manager.cc:1398] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:02.201334 11334 tablet_bootstrap.cc:492] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.201800 11340 leader_election.cc:288] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.202564 11333 tablet_bootstrap.cc:654] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.202927 11334 tablet_bootstrap.cc:654] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.214800 11334 tablet_bootstrap.cc:492] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.214991 11334 ts_tablet_manager.cc:1367] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20240503 14:12:02.215723 11334 raft_consensus.cc:348] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.215938 11334 raft_consensus.cc:374] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.216012 11334 raft_consensus.cc:729] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.216274 11334 consensus_queue.cc:260] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.216898 11334 ts_tablet_manager.cc:1398] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:02.217239 11334 tablet_bootstrap.cc:492] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.218999 11334 tablet_bootstrap.cc:654] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.223990 11332 tablet_bootstrap.cc:654] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.229708 11332 tablet_bootstrap.cc:492] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.230049 11332 ts_tablet_manager.cc:1367] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240503 14:12:02.231753 11332 raft_consensus.cc:348] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.232149 11332 raft_consensus.cc:374] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.232307 11332 raft_consensus.cc:729] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.232558 11340 raft_consensus.cc:482] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.233594 11340 raft_consensus.cc:504] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.234441 11340 leader_election.cc:288] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.232738 11332 consensus_queue.cc:260] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.235111 11332 ts_tablet_manager.cc:1398] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:02.235462 11332 tablet_bootstrap.cc:492] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.235613 11333 tablet_bootstrap.cc:492] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.235800 11333 ts_tablet_manager.cc:1367] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.019s
I20240503 14:12:02.235929 11334 tablet_bootstrap.cc:492] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.236086 11334 ts_tablet_manager.cc:1367] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.015s
I20240503 14:12:02.236580 11333 raft_consensus.cc:348] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.236812 11333 raft_consensus.cc:374] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.236876 11333 raft_consensus.cc:729] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.236943 11334 raft_consensus.cc:348] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.237155 11334 raft_consensus.cc:374] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.237154 11333 consensus_queue.cc:260] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.237259 11334 raft_consensus.cc:729] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.237847 11333 ts_tablet_manager.cc:1398] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:02.238204 11333 tablet_bootstrap.cc:492] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.238584 11334 consensus_queue.cc:260] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.239151 11334 ts_tablet_manager.cc:1398] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:02.240171 11334 tablet_bootstrap.cc:492] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.240393 11333 tablet_bootstrap.cc:654] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.240411 11332 tablet_bootstrap.cc:654] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.242962 11334 tablet_bootstrap.cc:654] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.246832 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9ed0f4fe8f4218b88e0ebec5034ae3" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.247413 11138 raft_consensus.cc:2454] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.247929 11137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac3f87f180472680a7f09b0ceef0b9" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.248236 11137 raft_consensus.cc:2454] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.248585 11338 raft_consensus.cc:482] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.248719 11338 raft_consensus.cc:504] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.248921 11333 tablet_bootstrap.cc:492] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.249089 11333 ts_tablet_manager.cc:1367] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 14:12:02.249430 11338 leader_election.cc:288] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.249933 11333 raft_consensus.cc:348] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.250130 11333 raft_consensus.cc:374] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.250231 11333 raft_consensus.cc:729] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.250207 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e834edfd67b54b6fbc62c3c848100708" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.250645 11269 raft_consensus.cc:2454] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfaac921af9f4b5dbc23321fbfb4d09e in term 0.
I20240503 14:12:02.250571 11333 consensus_queue.cc:260] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.251374 11333 ts_tablet_manager.cc:1398] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:02.251713 11333 tablet_bootstrap.cc:492] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.251698 11070 leader_election.cc:302] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.252914 11005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e834edfd67b54b6fbc62c3c848100708" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.253638 11333 tablet_bootstrap.cc:654] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.253670 11338 raft_consensus.cc:2781] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.253839 11338 raft_consensus.cc:482] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.253930 11338 raft_consensus.cc:3037] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:02.255051 11072 leader_election.cc:341] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.256033 11334 tablet_bootstrap.cc:492] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.256217 11334 ts_tablet_manager.cc:1367] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.002s
I20240503 14:12:02.256425 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac3f87f180472680a7f09b0ceef0b9" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.257156 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9ed0f4fe8f4218b88e0ebec5034ae3" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.257134 11334 raft_consensus.cc:348] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.257342 11268 raft_consensus.cc:2454] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.257460 11334 raft_consensus.cc:374] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.257556 11334 raft_consensus.cc:729] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.257450 10939 leader_election.cc:302] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.257807 11334 consensus_queue.cc:260] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
W20240503 14:12:02.258634 10938 leader_election.cc:341] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.258751 11340 raft_consensus.cc:2781] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.258891 11340 raft_consensus.cc:482] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.259007 10938 leader_election.cc:302] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, 85aecf73844f45aab998a795718f6824; no voters:
I20240503 14:12:02.259337 11334 ts_tablet_manager.cc:1398] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:12:02.259018 11340 raft_consensus.cc:3037] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.260324 11339 raft_consensus.cc:482] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.260483 11377 raft_consensus.cc:482] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.260493 11339 raft_consensus.cc:504] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.260720 11377 raft_consensus.cc:504] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.262221 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7730fcf05b144afb560055fb4802e57" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.262445 11268 raft_consensus.cc:2454] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfaac921af9f4b5dbc23321fbfb4d09e in term 0.
I20240503 14:12:02.262691 11339 leader_election.cc:288] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.264158 11005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7730fcf05b144afb560055fb4802e57" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
W20240503 14:12:02.265146 11072 leader_election.cc:341] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.269846 11377 leader_election.cc:288] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.272245 11378 raft_consensus.cc:2781] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.272367 11378 raft_consensus.cc:482] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.272509 11378 raft_consensus.cc:3037] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.273947 11332 tablet_bootstrap.cc:492] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.273996 11334 tablet_bootstrap.cc:492] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.274145 11332 ts_tablet_manager.cc:1367] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.004s
W20240503 14:12:02.274753 11184 tablet.cc:2369] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:02.275523 11332 raft_consensus.cc:348] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.275792 11332 raft_consensus.cc:374] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.275871 11332 raft_consensus.cc:729] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.276146 11332 consensus_queue.cc:260] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.276198 11070 leader_election.cc:302] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters: 85aecf73844f45aab998a795718f6824
I20240503 14:12:02.276724 11376 raft_consensus.cc:2781] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.276831 11376 raft_consensus.cc:482] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.276859 11381 raft_consensus.cc:482] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.276965 11376 raft_consensus.cc:3037] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.277001 11381 raft_consensus.cc:504] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.278219 11338 raft_consensus.cc:504] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.278323 11340 raft_consensus.cc:504] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.278921 11338 leader_election.cc:288] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 election: Requested vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.278946 11340 leader_election.cc:288] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Requested vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
W20240503 14:12:02.279515 11315 tablet.cc:2369] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:02.281183 11137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3a8130076746a88b31f2bd3ceab7c9" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.281574 11376 raft_consensus.cc:504] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.282269 11376 leader_election.cc:288] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 election: Requested vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.282514 11137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9ed0f4fe8f4218b88e0ebec5034ae3" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.282712 11137 raft_consensus.cc:3037] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.283004 11005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e834edfd67b54b6fbc62c3c848100708" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824"
I20240503 14:12:02.283303 11005 raft_consensus.cc:3037] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.283629 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e834edfd67b54b6fbc62c3c848100708" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.283854 11268 raft_consensus.cc:3037] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.283005 11004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7730fcf05b144afb560055fb4802e57" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824"
W20240503 14:12:02.285763 11052 tablet.cc:2369] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:02.286603 11339 raft_consensus.cc:482] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.286788 11339 raft_consensus.cc:504] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.287482 11339 leader_election.cc:288] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
W20240503 14:12:02.287575 10939 leader_election.cc:341] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.287786 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7730fcf05b144afb560055fb4802e57" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.287937 11334 tablet_bootstrap.cc:654] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.287971 11269 raft_consensus.cc:3037] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:02.288292 11072 leader_election.cc:341] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.289640 11381 leader_election.cc:288] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.290100 11333 tablet_bootstrap.cc:492] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.290056 11267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3a8130076746a88b31f2bd3ceab7c9" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.290308 11333 ts_tablet_manager.cc:1367] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.007s
I20240503 14:12:02.290405 11267 raft_consensus.cc:2454] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.290807 11266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9ed0f4fe8f4218b88e0ebec5034ae3" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.291206 11333 raft_consensus.cc:348] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.291474 11333 raft_consensus.cc:374] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.291575 11333 raft_consensus.cc:729] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.291580 10938 leader_election.cc:302] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, 85aecf73844f45aab998a795718f6824; no voters: bfaac921af9f4b5dbc23321fbfb4d09e
I20240503 14:12:02.291859 11333 consensus_queue.cc:260] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
W20240503 14:12:02.292120 10938 leader_election.cc:341] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.292390 11333 ts_tablet_manager.cc:1398] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:02.292702 11333 tablet_bootstrap.cc:492] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.293650 11382 raft_consensus.cc:2781] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.293745 11382 raft_consensus.cc:482] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.293864 11382 raft_consensus.cc:3037] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.294545 11332 ts_tablet_manager.cc:1398] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.020s user 0.003s sys 0.003s
I20240503 14:12:02.294873 11332 tablet_bootstrap.cc:492] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.296062 11333 tablet_bootstrap.cc:654] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.296178 11332 tablet_bootstrap.cc:654] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.300597 11333 tablet_bootstrap.cc:492] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.300894 11333 ts_tablet_manager.cc:1367] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240503 14:12:02.301930 11333 raft_consensus.cc:348] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.302258 11333 raft_consensus.cc:374] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.302397 11333 raft_consensus.cc:729] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.302770 11333 consensus_queue.cc:260] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.303506 11333 ts_tablet_manager.cc:1398] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:02.303941 11333 tablet_bootstrap.cc:492] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.304520 11332 tablet_bootstrap.cc:492] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.304684 11332 ts_tablet_manager.cc:1367] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240503 14:12:02.305703 11332 raft_consensus.cc:348] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.306067 11332 raft_consensus.cc:374] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.306100 11333 tablet_bootstrap.cc:654] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.306205 11332 raft_consensus.cc:729] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.306478 11332 consensus_queue.cc:260] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.306573 11376 raft_consensus.cc:482] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.306829 11376 raft_consensus.cc:504] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.307040 11332 ts_tablet_manager.cc:1398] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:02.307329 11332 tablet_bootstrap.cc:492] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.307512 11376 leader_election.cc:288] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.308509 11004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97ff554385741ada59c1d3779ef008f" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.309170 11332 tablet_bootstrap.cc:654] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:02.309209 11072 leader_election.cc:341] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.309837 11266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97ff554385741ada59c1d3779ef008f" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.310122 11266 raft_consensus.cc:2454] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfaac921af9f4b5dbc23321fbfb4d09e in term 0.
I20240503 14:12:02.310655 11070 leader_election.cc:302] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters: 85aecf73844f45aab998a795718f6824
I20240503 14:12:02.311417 11332 tablet_bootstrap.cc:492] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.312234 11332 ts_tablet_manager.cc:1367] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:02.313068 11332 raft_consensus.cc:348] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.313350 11332 raft_consensus.cc:374] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.313427 11332 raft_consensus.cc:729] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.313412 11376 raft_consensus.cc:2781] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.313513 11376 raft_consensus.cc:482] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.313779 11332 consensus_queue.cc:260] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.314297 11376 raft_consensus.cc:3037] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.314358 11332 ts_tablet_manager.cc:1398] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:02.314680 11332 tablet_bootstrap.cc:492] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.316515 11332 tablet_bootstrap.cc:654] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.318809 11333 tablet_bootstrap.cc:492] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.319083 11333 ts_tablet_manager.cc:1367] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20240503 14:12:02.320009 11333 raft_consensus.cc:348] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.320389 11333 raft_consensus.cc:374] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.320506 11333 raft_consensus.cc:729] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.320840 11333 consensus_queue.cc:260] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.321776 11004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204b25d73f784c2385cd2996d7fdaa5b" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.321890 11333 ts_tablet_manager.cc:1398] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:02.322026 11004 raft_consensus.cc:2454] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.322355 11333 tablet_bootstrap.cc:492] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.322407 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc770ed0b57a4a579abb38417866f3fb" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.322657 11003 raft_consensus.cc:2454] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.323271 11203 leader_election.cc:302] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, 85aecf73844f45aab998a795718f6824; no voters:
I20240503 14:12:02.323791 11203 leader_election.cc:302] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, 85aecf73844f45aab998a795718f6824; no voters:
I20240503 14:12:02.323866 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204b25d73f784c2385cd2996d7fdaa5b" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.324136 11138 raft_consensus.cc:2454] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.324692 11339 raft_consensus.cc:2781] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.324499 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc770ed0b57a4a579abb38417866f3fb" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.324785 11339 raft_consensus.cc:482] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.324892 11339 raft_consensus.cc:3037] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.324901 11333 tablet_bootstrap.cc:654] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
W20240503 14:12:02.326133 11202 leader_election.cc:341] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:02.327736 11334 tablet_bootstrap.cc:492] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.327894 11334 ts_tablet_manager.cc:1367] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.023s
I20240503 14:12:02.328370 11332 tablet_bootstrap.cc:492] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.328564 11332 ts_tablet_manager.cc:1367] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.001s
I20240503 14:12:02.328701 11334 raft_consensus.cc:348] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.328939 11334 raft_consensus.cc:374] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.329033 11334 raft_consensus.cc:729] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.329329 11334 consensus_queue.cc:260] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.329732 11332 raft_consensus.cc:348] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.329984 11332 raft_consensus.cc:374] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.330111 11332 raft_consensus.cc:729] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.330440 11332 consensus_queue.cc:260] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.331077 11332 ts_tablet_manager.cc:1398] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:02.331411 11332 tablet_bootstrap.cc:492] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.332618 11334 ts_tablet_manager.cc:1398] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:02.333248 11332 tablet_bootstrap.cc:654] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.333873 11389 raft_consensus.cc:2781] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.335642 11389 raft_consensus.cc:482] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.335866 11389 raft_consensus.cc:3037] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.336418 11334 tablet_bootstrap.cc:492] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.338012 11334 tablet_bootstrap.cc:654] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.338027 11332 tablet_bootstrap.cc:492] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.338215 11332 ts_tablet_manager.cc:1367] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:02.338372 11333 tablet_bootstrap.cc:492] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.338557 11333 ts_tablet_manager.cc:1367] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240503 14:12:02.338977 11332 raft_consensus.cc:348] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.339246 11332 raft_consensus.cc:374] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.339337 11332 raft_consensus.cc:729] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.339310 11333 raft_consensus.cc:348] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.339558 11333 raft_consensus.cc:374] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.339651 11333 raft_consensus.cc:729] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.339587 11332 consensus_queue.cc:260] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.339793 11334 tablet_bootstrap.cc:492] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.339946 11334 ts_tablet_manager.cc:1367] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:12:02.339890 11333 consensus_queue.cc:260] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.340142 11332 ts_tablet_manager.cc:1398] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:02.340508 11332 tablet_bootstrap.cc:492] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824: Bootstrap starting.
I20240503 14:12:02.340867 11333 ts_tablet_manager.cc:1398] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:02.340782 11334 raft_consensus.cc:348] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.341001 11334 raft_consensus.cc:374] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.341069 11334 raft_consensus.cc:729] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.341130 11333 tablet_bootstrap.cc:492] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.341346 11334 consensus_queue.cc:260] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.342794 11332 tablet_bootstrap.cc:654] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.343262 11334 ts_tablet_manager.cc:1398] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:02.343605 11334 tablet_bootstrap.cc:492] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.343712 11333 tablet_bootstrap.cc:654] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.345142 11334 tablet_bootstrap.cc:654] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.350378 11334 tablet_bootstrap.cc:492] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.350549 11334 ts_tablet_manager.cc:1367] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:12:02.350764 11332 tablet_bootstrap.cc:492] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824: No bootstrap required, opened a new log
I20240503 14:12:02.350924 11332 ts_tablet_manager.cc:1367] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240503 14:12:02.351326 11333 tablet_bootstrap.cc:492] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.351497 11333 ts_tablet_manager.cc:1367] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20240503 14:12:02.351465 11334 raft_consensus.cc:348] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.351662 11334 raft_consensus.cc:374] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.351773 11334 raft_consensus.cc:729] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.351823 11332 raft_consensus.cc:348] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.352033 11332 raft_consensus.cc:374] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.352128 11332 raft_consensus.cc:729] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aecf73844f45aab998a795718f6824, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.352049 11334 consensus_queue.cc:260] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.352279 11333 raft_consensus.cc:348] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.352396 11332 consensus_queue.cc:260] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.352497 11333 raft_consensus.cc:374] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.352561 11333 raft_consensus.cc:729] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.352820 11381 raft_consensus.cc:482] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.352855 11333 consensus_queue.cc:260] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.353075 11332 ts_tablet_manager.cc:1398] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:02.353145 11334 ts_tablet_manager.cc:1398] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:02.353029 11381 raft_consensus.cc:504] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.353382 11333 ts_tablet_manager.cc:1398] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:02.353626 11333 tablet_bootstrap.cc:492] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce: Bootstrap starting.
I20240503 14:12:02.353745 11381 leader_election.cc:288] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.354310 11334 tablet_bootstrap.cc:492] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.354305 11266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e12492b6da43abab399c40c0232c9d" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.354343 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e12492b6da43abab399c40c0232c9d" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.354545 11136 raft_consensus.cc:2454] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.355093 10939 leader_election.cc:302] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.355357 11381 raft_consensus.cc:2781] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.355338 11266 raft_consensus.cc:2454] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.355430 11381 raft_consensus.cc:482] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.355516 11381 raft_consensus.cc:3037] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.355974 11334 tablet_bootstrap.cc:654] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.356633 11333 tablet_bootstrap.cc:654] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.358640 11333 tablet_bootstrap.cc:492] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce: No bootstrap required, opened a new log
I20240503 14:12:02.358754 11333 ts_tablet_manager.cc:1367] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240503 14:12:02.359329 11333 raft_consensus.cc:348] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.359480 11333 raft_consensus.cc:374] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.359532 11333 raft_consensus.cc:729] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.359694 11333 consensus_queue.cc:260] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.360121 11333 ts_tablet_manager.cc:1398] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:12:02.361258 11334 tablet_bootstrap.cc:492] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.361440 11334 ts_tablet_manager.cc:1367] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 14:12:02.362037 11334 raft_consensus.cc:348] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.362216 11334 raft_consensus.cc:374] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.362280 11334 raft_consensus.cc:729] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.362502 11334 consensus_queue.cc:260] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.362928 11334 ts_tablet_manager.cc:1398] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:12:02.363775 11334 tablet_bootstrap.cc:492] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e: Bootstrap starting.
I20240503 14:12:02.365159 11334 tablet_bootstrap.cc:654] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:02.366914 11334 tablet_bootstrap.cc:492] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e: No bootstrap required, opened a new log
I20240503 14:12:02.367048 11334 ts_tablet_manager.cc:1367] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:02.367707 11334 raft_consensus.cc:348] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } }
I20240503 14:12:02.367895 11334 raft_consensus.cc:374] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:02.367982 11334 raft_consensus.cc:729] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Initialized, Role: FOLLOWER
I20240503 14:12:02.368167 11334 consensus_queue.cc:260] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } }
I20240503 14:12:02.368624 11334 ts_tablet_manager.cc:1398] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:12:02.375409 11340 raft_consensus.cc:482] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.375620 11340 raft_consensus.cc:504] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.376224 11340 leader_election.cc:288] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.376691 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90da99a633945fcaf6a1740d64e6d50" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.376775 11266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90da99a633945fcaf6a1740d64e6d50" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.376942 11136 raft_consensus.cc:2454] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.376996 11266 raft_consensus.cc:2454] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.377388 10939 leader_election.cc:302] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.377664 11340 raft_consensus.cc:2781] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.377764 11340 raft_consensus.cc:482] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.377843 11340 raft_consensus.cc:3037] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.380143 11137 raft_consensus.cc:2454] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.380163 11268 raft_consensus.cc:2454] T e834edfd67b54b6fbc62c3c848100708 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfaac921af9f4b5dbc23321fbfb4d09e in term 1.
I20240503 14:12:02.380156 11378 raft_consensus.cc:504] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.380838 10939 leader_election.cc:302] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters: 1eaac79055b14e99b0f6d1097cae21ce
I20240503 14:12:02.381119 11005 raft_consensus.cc:2454] T e834edfd67b54b6fbc62c3c848100708 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfaac921af9f4b5dbc23321fbfb4d09e in term 1.
I20240503 14:12:02.380861 11378 leader_election.cc:288] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Requested vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.381421 11378 raft_consensus.cc:2781] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.381572 11378 raft_consensus.cc:686] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [term 1 LEADER]: Becoming Leader. State: Replica: 85aecf73844f45aab998a795718f6824, State: Running, Role: LEADER
I20240503 14:12:02.382164 11378 consensus_queue.cc:237] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.382891 11376 raft_consensus.cc:504] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.383980 11382 raft_consensus.cc:504] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.384601 11382 leader_election.cc:288] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Requested vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.387338 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac3f87f180472680a7f09b0ceef0b9" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.387610 11268 raft_consensus.cc:3037] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.388010 11266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3a8130076746a88b31f2bd3ceab7c9" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.388214 11266 raft_consensus.cc:3037] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.388896 11072 leader_election.cc:302] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.389307 11338 raft_consensus.cc:2781] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.389497 11338 raft_consensus.cc:686] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 LEADER]: Becoming Leader. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Running, Role: LEADER
I20240503 14:12:02.389772 11005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97ff554385741ada59c1d3779ef008f" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824"
I20240503 14:12:02.390018 11005 raft_consensus.cc:3037] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.390039 11338 consensus_queue.cc:237] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.390605 11376 leader_election.cc:288] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 election: Requested vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.393482 11393 raft_consensus.cc:482] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.393661 11393 raft_consensus.cc:504] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.394594 11393 leader_election.cc:288] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.395439 11137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e92a0269bf8444bb814daba5ea0c701" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.395660 11137 raft_consensus.cc:2454] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.395678 11267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97ff554385741ada59c1d3779ef008f" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.395849 11267 raft_consensus.cc:3037] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.396147 11202 leader_election.cc:302] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.396441 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e92a0269bf8444bb814daba5ea0c701" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.396667 11003 raft_consensus.cc:2454] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.397097 11393 raft_consensus.cc:2781] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.397231 11393 raft_consensus.cc:482] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.397343 11393 raft_consensus.cc:3037] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.402392 11137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac3f87f180472680a7f09b0ceef0b9" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.402500 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3a8130076746a88b31f2bd3ceab7c9" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.402644 11137 raft_consensus.cc:3037] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.402736 11136 raft_consensus.cc:3037] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.401338 10862 catalog_manager.cc:5484] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e reported cstate change: term changed from 0 to 1, leader changed from <none> to bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2). New cstate: current_term: 1 leader_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:02.422495 11400 raft_consensus.cc:482] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.422726 11400 raft_consensus.cc:504] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.423529 11400 leader_election.cc:288] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.424104 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6ce01ac44f4b4a8aac787f03d83af3" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.424132 11265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6ce01ac44f4b4a8aac787f03d83af3" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.424335 11138 raft_consensus.cc:2454] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.424417 11265 raft_consensus.cc:2454] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.424741 10939 leader_election.cc:302] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.425040 11400 raft_consensus.cc:2781] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.425165 11400 raft_consensus.cc:482] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.425246 11400 raft_consensus.cc:3037] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.426717 11406 raft_consensus.cc:482] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.426864 11406 raft_consensus.cc:504] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.427541 11406 leader_election.cc:288] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.428088 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554b725e046a4dbbae9b4a6126c95389" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.428097 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554b725e046a4dbbae9b4a6126c95389" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.428277 11003 raft_consensus.cc:2454] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.428318 11138 raft_consensus.cc:2454] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.428726 11203 leader_election.cc:302] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, 85aecf73844f45aab998a795718f6824; no voters:
I20240503 14:12:02.429035 11406 raft_consensus.cc:2781] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.429158 11406 raft_consensus.cc:482] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.429235 11406 raft_consensus.cc:3037] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.447268 11407 raft_consensus.cc:482] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.447510 11407 raft_consensus.cc:504] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.448429 11407 leader_election.cc:288] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899)
I20240503 14:12:02.449069 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c7203d4650496188b9d84858bfcc35" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.449077 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c7203d4650496188b9d84858bfcc35" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.449333 11138 raft_consensus.cc:2454] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.449334 11003 raft_consensus.cc:2454] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.449800 11202 leader_election.cc:302] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.450138 11407 raft_consensus.cc:2781] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.450259 11407 raft_consensus.cc:482] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.450321 11407 raft_consensus.cc:3037] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.454818 11408 raft_consensus.cc:482] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.455014 11408 raft_consensus.cc:504] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.455807 11408 leader_election.cc:288] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.456687 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06166dc9654d408193d9734634654673" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.456687 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06166dc9654d408193d9734634654673" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.456934 11003 raft_consensus.cc:2454] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.456934 11138 raft_consensus.cc:2454] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 0.
I20240503 14:12:02.457512 11203 leader_election.cc:302] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, 85aecf73844f45aab998a795718f6824; no voters:
I20240503 14:12:02.457800 11408 raft_consensus.cc:2781] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.457979 11408 raft_consensus.cc:482] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.458074 11408 raft_consensus.cc:3037] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.477046 11378 raft_consensus.cc:482] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.477242 11378 raft_consensus.cc:504] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } }
I20240503 14:12:02.477947 11378 leader_election.cc:288] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.478528 11265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5c86d427064ebc803628e9269f4fe0" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.478528 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5c86d427064ebc803628e9269f4fe0" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.478731 11265 raft_consensus.cc:2454] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.478746 11138 raft_consensus.cc:2454] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 0.
I20240503 14:12:02.479118 11137 raft_consensus.cc:2454] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.479128 11269 raft_consensus.cc:2454] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfaac921af9f4b5dbc23321fbfb4d09e in term 1.
I20240503 14:12:02.479511 11136 raft_consensus.cc:2454] T 5c3a8130076746a88b31f2bd3ceab7c9 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.479902 11381 raft_consensus.cc:504] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.480609 11381 leader_election.cc:288] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Requested vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.481963 11339 raft_consensus.cc:504] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.482748 11339 leader_election.cc:288] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Requested vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.483376 10939 leader_election.cc:302] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.483520 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204b25d73f784c2385cd2996d7fdaa5b" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824"
I20240503 14:12:02.483716 11003 raft_consensus.cc:3037] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.484377 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e12492b6da43abab399c40c0232c9d" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.484483 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e12492b6da43abab399c40c0232c9d" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.484594 11136 raft_consensus.cc:3037] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.484694 11269 raft_consensus.cc:3037] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.484678 10939 leader_election.cc:302] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.484938 11070 leader_election.cc:302] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters: 85aecf73844f45aab998a795718f6824
I20240503 14:12:02.485019 10939 leader_election.cc:302] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.485329 11378 raft_consensus.cc:2781] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.483712 11381 raft_consensus.cc:2781] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.485425 11338 raft_consensus.cc:2781] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.485502 11381 raft_consensus.cc:482] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.485574 11338 raft_consensus.cc:686] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 LEADER]: Becoming Leader. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Running, Role: LEADER
I20240503 14:12:02.485610 11381 raft_consensus.cc:3037] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.485859 11338 consensus_queue.cc:237] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.486105 11378 raft_consensus.cc:686] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [term 1 LEADER]: Becoming Leader. State: Replica: 85aecf73844f45aab998a795718f6824, State: Running, Role: LEADER
I20240503 14:12:02.486299 11409 raft_consensus.cc:2781] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.486450 11378 consensus_queue.cc:237] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.487653 11137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204b25d73f784c2385cd2996d7fdaa5b" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.487928 11137 raft_consensus.cc:3037] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.489043 11409 raft_consensus.cc:686] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [term 1 LEADER]: Becoming Leader. State: Replica: 85aecf73844f45aab998a795718f6824, State: Running, Role: LEADER
I20240503 14:12:02.489451 11409 consensus_queue.cc:237] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.489923 11338 raft_consensus.cc:482] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.490077 11338 raft_consensus.cc:504] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.490232 10862 catalog_manager.cc:5484] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e reported cstate change: term changed from 0 to 1, leader changed from <none> to bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2). New cstate: current_term: 1 leader_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:02.490777 11338 leader_election.cc:288] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.491189 11004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc770ed0b57a4a579abb38417866f3fb" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.491402 11004 raft_consensus.cc:2454] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfaac921af9f4b5dbc23321fbfb4d09e in term 0.
I20240503 14:12:02.491468 11265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc770ed0b57a4a579abb38417866f3fb" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.491859 11072 leader_election.cc:302] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.492120 11338 raft_consensus.cc:2781] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:02.492216 11338 raft_consensus.cc:482] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:02.492296 11338 raft_consensus.cc:3037] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.579573 11136 raft_consensus.cc:2454] T 28e12492b6da43abab399c40c0232c9d P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.579553 11268 raft_consensus.cc:2454] T 26ac3f87f180472680a7f09b0ceef0b9 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.579939 11340 raft_consensus.cc:504] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.580791 11340 leader_election.cc:288] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Requested vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.582633 11137 raft_consensus.cc:2454] T 204b25d73f784c2385cd2996d7fdaa5b P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 1.
I20240503 14:12:02.582859 11267 raft_consensus.cc:2454] T c97ff554385741ada59c1d3779ef008f P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfaac921af9f4b5dbc23321fbfb4d09e in term 1.
I20240503 14:12:02.583879 10939 leader_election.cc:302] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.584190 11202 leader_election.cc:302] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.584386 11267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90da99a633945fcaf6a1740d64e6d50" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.584635 11267 raft_consensus.cc:3037] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.584813 11070 leader_election.cc:302] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.585455 11340 raft_consensus.cc:2781] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.585780 11340 raft_consensus.cc:686] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [term 1 LEADER]: Becoming Leader. State: Replica: 85aecf73844f45aab998a795718f6824, State: Running, Role: LEADER
I20240503 14:12:02.585961 11339 raft_consensus.cc:2781] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.586374 11376 raft_consensus.cc:2781] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.586544 11376 raft_consensus.cc:686] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 LEADER]: Becoming Leader. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Running, Role: LEADER
I20240503 14:12:02.586863 11376 consensus_queue.cc:237] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.587213 11137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90da99a633945fcaf6a1740d64e6d50" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.587641 11137 raft_consensus.cc:3037] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.590956 11339 raft_consensus.cc:686] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [term 1 LEADER]: Becoming Leader. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Running, Role: LEADER
I20240503 14:12:02.592211 11339 consensus_queue.cc:237] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.592343 11340 consensus_queue.cc:237] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.598206 10862 catalog_manager.cc:5484] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3). New cstate: current_term: 1 leader_uuid: "1eaac79055b14e99b0f6d1097cae21ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:02.671483 11003 raft_consensus.cc:2454] T 204b25d73f784c2385cd2996d7fdaa5b P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 1.
I20240503 14:12:02.671483 11338 raft_consensus.cc:504] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.671517 11393 raft_consensus.cc:504] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.672286 11393 leader_election.cc:288] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Requested vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.672331 11338 leader_election.cc:288] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 election: Requested vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.672714 11137 raft_consensus.cc:2454] T d90da99a633945fcaf6a1740d64e6d50 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.674906 11407 raft_consensus.cc:504] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.675540 11407 leader_election.cc:288] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Requested vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899)
I20240503 14:12:02.676003 11266 raft_consensus.cc:2454] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.676476 11389 raft_consensus.cc:504] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.677227 11389 leader_election.cc:288] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Requested vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.678332 10939 leader_election.cc:302] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.678627 11340 raft_consensus.cc:2781] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.678786 11340 raft_consensus.cc:686] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [term 1 LEADER]: Becoming Leader. State: Replica: 85aecf73844f45aab998a795718f6824, State: Running, Role: LEADER
I20240503 14:12:02.679113 11340 consensus_queue.cc:237] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.679530 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc770ed0b57a4a579abb38417866f3fb" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824"
I20240503 14:12:02.679805 11003 raft_consensus.cc:3037] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.680281 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c7203d4650496188b9d84858bfcc35" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.680488 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc770ed0b57a4a579abb38417866f3fb" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.680528 11136 raft_consensus.cc:3037] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.680853 11138 raft_consensus.cc:2379] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1eaac79055b14e99b0f6d1097cae21ce in current term 1: Already voted for candidate bfaac921af9f4b5dbc23321fbfb4d09e in this term.
I20240503 14:12:02.682979 11265 raft_consensus.cc:2379] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bfaac921af9f4b5dbc23321fbfb4d09e in current term 1: Already voted for candidate 1eaac79055b14e99b0f6d1097cae21ce in this term.
I20240503 14:12:02.680526 11137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e92a0269bf8444bb814daba5ea0c701" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.684360 11004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e92a0269bf8444bb814daba5ea0c701" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824"
I20240503 14:12:02.684479 11137 raft_consensus.cc:3037] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.684590 11004 raft_consensus.cc:3037] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.684553 11002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89c7203d4650496188b9d84858bfcc35" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824"
I20240503 14:12:02.684856 11002 raft_consensus.cc:3037] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.685382 11001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc770ed0b57a4a579abb38417866f3fb" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824"
I20240503 14:12:02.685621 11265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc770ed0b57a4a579abb38417866f3fb" candidate_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.685812 11265 raft_consensus.cc:2379] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bfaac921af9f4b5dbc23321fbfb4d09e in current term 1: Already voted for candidate 1eaac79055b14e99b0f6d1097cae21ce in this term.
I20240503 14:12:02.686431 11203 leader_election.cc:302] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce; no voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e
I20240503 14:12:02.686793 11389 raft_consensus.cc:2726] T fc770ed0b57a4a579abb38417866f3fb P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.688114 10862 catalog_manager.cc:5484] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e reported cstate change: term changed from 0 to 1, leader changed from <none> to bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2). New cstate: current_term: 1 leader_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:02.730530 11340 raft_consensus.cc:482] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.730785 11340 raft_consensus.cc:504] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.731560 11340 leader_election.cc:288] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.732218 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7730fcf05b144afb560055fb4802e57" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.732264 11265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7730fcf05b144afb560055fb4802e57" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce" is_pre_election: true
I20240503 14:12:02.732517 11265 raft_consensus.cc:2379] T b7730fcf05b144afb560055fb4802e57 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85aecf73844f45aab998a795718f6824 in current term 1: Already voted for candidate bfaac921af9f4b5dbc23321fbfb4d09e in this term.
I20240503 14:12:02.733033 10938 leader_election.cc:302] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85aecf73844f45aab998a795718f6824; no voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e
I20240503 14:12:02.733422 11340 raft_consensus.cc:3037] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.736213 11389 raft_consensus.cc:482] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:02.736421 11389 raft_consensus.cc:504] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.737248 11389 leader_election.cc:288] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.737825 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9ed0f4fe8f4218b88e0ebec5034ae3" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" is_pre_election: true
I20240503 14:12:02.738118 11138 raft_consensus.cc:2379] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1eaac79055b14e99b0f6d1097cae21ce in current term 1: Already voted for candidate 85aecf73844f45aab998a795718f6824 in this term.
I20240503 14:12:02.738092 11001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9ed0f4fe8f4218b88e0ebec5034ae3" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824" is_pre_election: true
I20240503 14:12:02.738700 11203 leader_election.cc:302] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce; no voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e
I20240503 14:12:02.738973 11389 raft_consensus.cc:3037] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.768728 11136 raft_consensus.cc:2454] T 89c7203d4650496188b9d84858bfcc35 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 1.
I20240503 14:12:02.768765 11381 raft_consensus.cc:504] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } }
I20240503 14:12:02.769585 11381 leader_election.cc:288] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Requested vote from peers 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.769261 11408 raft_consensus.cc:504] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.770038 11400 raft_consensus.cc:504] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.770259 11408 leader_election.cc:288] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Requested vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.770699 11400 leader_election.cc:288] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Requested vote from peers bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853), 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3:33607)
I20240503 14:12:02.771042 11005 raft_consensus.cc:2454] T c97ff554385741ada59c1d3779ef008f P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfaac921af9f4b5dbc23321fbfb4d09e in term 1.
I20240503 14:12:02.774991 11266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6ce01ac44f4b4a8aac787f03d83af3" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.775221 11266 raft_consensus.cc:3037] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.775322 11202 leader_election.cc:302] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.775969 11135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06166dc9654d408193d9734634654673" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.776031 11265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5c86d427064ebc803628e9269f4fe0" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaac79055b14e99b0f6d1097cae21ce"
I20240503 14:12:02.776087 11005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06166dc9654d408193d9734634654673" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824"
I20240503 14:12:02.776186 11135 raft_consensus.cc:3037] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.776209 11265 raft_consensus.cc:3037] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.776290 11005 raft_consensus.cc:3037] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.776648 11138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6ce01ac44f4b4a8aac787f03d83af3" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.776880 11138 raft_consensus.cc:3037] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.776883 11408 raft_consensus.cc:2781] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.777007 11408 raft_consensus.cc:686] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 LEADER]: Becoming Leader. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Running, Role: LEADER
I20240503 14:12:02.776643 11136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5c86d427064ebc803628e9269f4fe0" candidate_uuid: "85aecf73844f45aab998a795718f6824" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.777344 11408 consensus_queue.cc:237] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.777771 11136 raft_consensus.cc:3037] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.780417 10862 catalog_manager.cc:5484] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3). New cstate: current_term: 1 leader_uuid: "1eaac79055b14e99b0f6d1097cae21ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:02.867115 11269 raft_consensus.cc:2454] T 28e12492b6da43abab399c40c0232c9d P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.867115 11003 raft_consensus.cc:2454] T fc770ed0b57a4a579abb38417866f3fb P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfaac921af9f4b5dbc23321fbfb4d09e in term 1.
I20240503 14:12:02.867599 11137 raft_consensus.cc:2454] T 3e92a0269bf8444bb814daba5ea0c701 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 1.
I20240503 14:12:02.867692 11406 raft_consensus.cc:504] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.868435 11406 leader_election.cc:288] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Requested vote from peers 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899), bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2:36853)
I20240503 14:12:02.870522 11135 raft_consensus.cc:2454] T 06166dc9654d408193d9734634654673 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 1.
I20240503 14:12:02.873296 11202 leader_election.cc:302] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.873725 11202 leader_election.cc:302] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.874589 11408 raft_consensus.cc:2781] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.874841 11407 raft_consensus.cc:2781] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.874981 11407 raft_consensus.cc:686] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 LEADER]: Becoming Leader. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Running, Role: LEADER
I20240503 14:12:02.875299 11407 consensus_queue.cc:237] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.876432 11408 raft_consensus.cc:686] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 LEADER]: Becoming Leader. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Running, Role: LEADER
I20240503 14:12:02.876780 11408 consensus_queue.cc:237] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.878280 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554b725e046a4dbbae9b4a6126c95389" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85aecf73844f45aab998a795718f6824"
I20240503 14:12:02.878557 11003 raft_consensus.cc:3037] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.879004 11072 leader_election.cc:302] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters: 1eaac79055b14e99b0f6d1097cae21ce
I20240503 14:12:02.879746 11338 raft_consensus.cc:2781] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.879894 11338 raft_consensus.cc:686] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 LEADER]: Becoming Leader. State: Replica: bfaac921af9f4b5dbc23321fbfb4d09e, State: Running, Role: LEADER
I20240503 14:12:02.880172 11338 consensus_queue.cc:237] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.881422 11135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "554b725e046a4dbbae9b4a6126c95389" candidate_uuid: "1eaac79055b14e99b0f6d1097cae21ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e"
I20240503 14:12:02.881672 11135 raft_consensus.cc:3037] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:02.886411 11400 consensus_queue.cc:1035] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:02.889962 11376 consensus_queue.cc:1035] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:02.892401 11400 consensus_queue.cc:1035] T 26ac3f87f180472680a7f09b0ceef0b9 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:02.893095 10863 catalog_manager.cc:5484] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e reported cstate change: term changed from 0 to 1, leader changed from <none> to bfaac921af9f4b5dbc23321fbfb4d09e (127.10.145.2). New cstate: current_term: 1 leader_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:02.893426 10862 catalog_manager.cc:5484] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3). New cstate: current_term: 1 leader_uuid: "1eaac79055b14e99b0f6d1097cae21ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:02.894050 10862 catalog_manager.cc:5484] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3). New cstate: current_term: 1 leader_uuid: "1eaac79055b14e99b0f6d1097cae21ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:02.919075 10863 catalog_manager.cc:5484] T 26ac3f87f180472680a7f09b0ceef0b9 P bfaac921af9f4b5dbc23321fbfb4d09e reported cstate change: term changed from 0 to 1, leader changed from <none> to 85aecf73844f45aab998a795718f6824 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "85aecf73844f45aab998a795718f6824" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } }
I20240503 14:12:02.921428 11338 consensus_queue.cc:1035] T e834edfd67b54b6fbc62c3c848100708 P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:02.945827 11400 consensus_queue.cc:1035] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:02.949146 10862 catalog_manager.cc:5484] T 5c3a8130076746a88b31f2bd3ceab7c9 P 1eaac79055b14e99b0f6d1097cae21ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 85aecf73844f45aab998a795718f6824 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "85aecf73844f45aab998a795718f6824" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } }
I20240503 14:12:02.950647 11381 consensus_queue.cc:1035] T 5c3a8130076746a88b31f2bd3ceab7c9 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:02.966516 11138 raft_consensus.cc:2454] T 6f6ce01ac44f4b4a8aac787f03d83af3 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.966531 11002 raft_consensus.cc:2454] T 89c7203d4650496188b9d84858bfcc35 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 1.
I20240503 14:12:02.966517 11267 raft_consensus.cc:2454] T d90da99a633945fcaf6a1740d64e6d50 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.967070 11005 raft_consensus.cc:2454] T 06166dc9654d408193d9734634654673 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 1.
I20240503 14:12:02.967109 10939 leader_election.cc:302] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:02.967437 11381 raft_consensus.cc:2781] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.967576 11381 raft_consensus.cc:686] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [term 1 LEADER]: Becoming Leader. State: Replica: 85aecf73844f45aab998a795718f6824, State: Running, Role: LEADER
I20240503 14:12:02.967742 11004 raft_consensus.cc:2454] T 3e92a0269bf8444bb814daba5ea0c701 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 1.
I20240503 14:12:02.967854 11381 consensus_queue.cc:237] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:02.969483 11266 raft_consensus.cc:2454] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.969939 11389 raft_consensus.cc:2726] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:02.970739 11265 raft_consensus.cc:2454] T 9f5c86d427064ebc803628e9269f4fe0 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:02.971879 10938 leader_election.cc:302] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, 85aecf73844f45aab998a795718f6824; no voters:
I20240503 14:12:02.972167 11381 raft_consensus.cc:2781] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:02.972304 11381 raft_consensus.cc:686] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [term 1 LEADER]: Becoming Leader. State: Replica: 85aecf73844f45aab998a795718f6824, State: Running, Role: LEADER
I20240503 14:12:02.972393 11409 consensus_queue.cc:1035] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:02.972570 11381 consensus_queue.cc:237] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } }
I20240503 14:12:02.975854 10863 catalog_manager.cc:5484] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 1eaac79055b14e99b0f6d1097cae21ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 85aecf73844f45aab998a795718f6824 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "85aecf73844f45aab998a795718f6824" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } } }
I20240503 14:12:02.977106 11381 consensus_queue.cc:1035] T 7a9ed0f4fe8f4218b88e0ebec5034ae3 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:02.980329 11338 consensus_queue.cc:1035] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:12:02.984663 11376 consensus_queue.cc:1035] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:02.989924 11338 consensus_queue.cc:1035] T c97ff554385741ada59c1d3779ef008f P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.064834 11340 raft_consensus.cc:2726] T b7730fcf05b144afb560055fb4802e57 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:03.064837 11136 raft_consensus.cc:2454] T 9f5c86d427064ebc803628e9269f4fe0 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aecf73844f45aab998a795718f6824 in term 1.
I20240503 14:12:03.067633 11135 raft_consensus.cc:2454] T 554b725e046a4dbbae9b4a6126c95389 P bfaac921af9f4b5dbc23321fbfb4d09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 1.
I20240503 14:12:03.067819 11003 raft_consensus.cc:2454] T 554b725e046a4dbbae9b4a6126c95389 P 85aecf73844f45aab998a795718f6824 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaac79055b14e99b0f6d1097cae21ce in term 1.
I20240503 14:12:03.068421 11202 leader_election.cc:302] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1eaac79055b14e99b0f6d1097cae21ce, 85aecf73844f45aab998a795718f6824, bfaac921af9f4b5dbc23321fbfb4d09e; no voters:
I20240503 14:12:03.068787 11389 raft_consensus.cc:2781] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:03.068959 11389 raft_consensus.cc:686] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [term 1 LEADER]: Becoming Leader. State: Replica: 1eaac79055b14e99b0f6d1097cae21ce, State: Running, Role: LEADER
I20240503 14:12:03.069226 11389 consensus_queue.cc:237] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } }
I20240503 14:12:03.069540 11338 consensus_queue.cc:1035] T b7730fcf05b144afb560055fb4802e57 P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.072475 11389 consensus_queue.cc:1035] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.075479 10863 catalog_manager.cc:5484] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eaac79055b14e99b0f6d1097cae21ce (127.10.145.3). New cstate: current_term: 1 leader_uuid: "1eaac79055b14e99b0f6d1097cae21ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:03.079751 10862 catalog_manager.cc:5484] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85aecf73844f45aab998a795718f6824 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "85aecf73844f45aab998a795718f6824" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:03.080250 10862 catalog_manager.cc:5484] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85aecf73844f45aab998a795718f6824 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "85aecf73844f45aab998a795718f6824" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:03.080714 10862 catalog_manager.cc:5484] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85aecf73844f45aab998a795718f6824 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "85aecf73844f45aab998a795718f6824" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:03.081064 10862 catalog_manager.cc:5484] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85aecf73844f45aab998a795718f6824 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "85aecf73844f45aab998a795718f6824" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:03.107298 11406 consensus_queue.cc:1035] T 204b25d73f784c2385cd2996d7fdaa5b P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.136390 11340 consensus_queue.cc:1035] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.141425 11410 consensus_queue.cc:1035] T 28e12492b6da43abab399c40c0232c9d P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.227829 11410 consensus_queue.cc:1035] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.232677 11340 consensus_queue.cc:1035] T d90da99a633945fcaf6a1740d64e6d50 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.307330 11406 consensus_queue.cc:1035] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.307269 11389 consensus_queue.cc:1035] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.315784 11408 consensus_queue.cc:1035] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.316566 11406 consensus_queue.cc:1035] T 3e92a0269bf8444bb814daba5ea0c701 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.348161 11338 consensus_queue.cc:1035] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.353699 11338 consensus_queue.cc:1035] T fc770ed0b57a4a579abb38417866f3fb P bfaac921af9f4b5dbc23321fbfb4d09e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.358953 11340 consensus_queue.cc:1035] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.373315 11340 consensus_queue.cc:1035] T 9f5c86d427064ebc803628e9269f4fe0 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.384584 11389 consensus_queue.cc:1035] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.400619 11389 consensus_queue.cc:1035] T 06166dc9654d408193d9734634654673 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.411635 11340 consensus_queue.cc:1035] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaac79055b14e99b0f6d1097cae21ce" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.417258 11340 consensus_queue.cc:1035] T 6f6ce01ac44f4b4a8aac787f03d83af3 P 85aecf73844f45aab998a795718f6824 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.450970 11389 consensus_queue.cc:1035] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaac921af9f4b5dbc23321fbfb4d09e" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:03.456350 11408 consensus_queue.cc:1035] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aecf73844f45aab998a795718f6824" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:03.717406 11447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:03.717761 11447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:03.725417 11447 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 380.241 ms
time per row: 0.19012 ms
I20240503 14:12:04.787896 11244 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:04.788214 10978 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:04.790362 11117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:04.971815 10978 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:04.972374 11244 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:04.987166 11117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:05.199329 11244 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:05.201833 10978 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:05.201828 11117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:05.868691 11244 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:05.873052 11117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:05.875463 10978 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:06.057143 11244 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:06.063023 11117 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:06.073484 10978 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:06.216859 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 10923
I20240503 14:12:06.223582 11046 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:06.348832 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 10923
W20240503 14:12:06.366353 11072 connection.cc:535] server connection from 127.10.145.1:44711 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:06.366820 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 11055
I20240503 14:12:06.373014 11178 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:12:06.418962 11203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.1:32899: connect: Connection refused (error 111)
W20240503 14:12:06.420599 11203 consensus_peers.cc:485] T 554b725e046a4dbbae9b4a6126c95389 P 1eaac79055b14e99b0f6d1097cae21ce -> Peer 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899): Couldn't send request to peer 85aecf73844f45aab998a795718f6824. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:32899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:06.428805 11203 consensus_peers.cc:485] T 89c7203d4650496188b9d84858bfcc35 P 1eaac79055b14e99b0f6d1097cae21ce -> Peer 85aecf73844f45aab998a795718f6824 (127.10.145.1:32899): Couldn't send request to peer 85aecf73844f45aab998a795718f6824. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:32899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:06.494195 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 11055
W20240503 14:12:06.511572 11203 connection.cc:535] server connection from 127.10.145.2:57287 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:06.511801 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 11186
I20240503 14:12:06.517762 11309 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:06.642446 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 11186
I20240503 14:12:06.660727 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 10832
I20240503 14:12:06.661830 10894 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:06.781082 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 10832
2024-05-03T14:12:06Z chronyd exiting
[ OK ] TableRangeRebalancingTest.Basic (10953 ms)
[----------] 1 test from TableRangeRebalancingTest (10953 ms total)
[----------] 1 test from RebalanceStartSafetyTest
[ RUN ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1
I20240503 14:12:06.829411 10820 test_util.cc:268] Using random seed: 1308737869
I20240503 14:12:06.831249 10820 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:12:06.831319 10820 ts_itest-base.cc:113] --------------
I20240503 14:12:06.831377 10820 ts_itest-base.cc:114] 5 tablet servers
I20240503 14:12:06.831415 10820 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:12:06.831475 10820 ts_itest-base.cc:116] --------------
2024-05-03T14:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:12:06Z Disabled control of system clock
I20240503 14:12:06.853165 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:41805
--webserver_interface=127.10.145.62
--webserver_port=0
--builtin_ntp_servers=127.10.145.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.145.62:41805
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 14:12:07.192668 11710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:07.192991 11710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:07.193113 11710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:07.200721 11710 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:07.200848 11710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:07.200918 11710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:07.201007 11710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:07.201076 11710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:07.210475 11710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.10.145.62:41805
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:41805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:07.211939 11710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:07.214067 11710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:07.225734 11718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.226104 11715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.226470 11716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.226938 11710 server_base.cc:1034] running on GCE node
I20240503 14:12:07.227759 11710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:07.229326 11710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:07.230556 11710 hybrid_clock.cc:648] HybridClock initialized: now 1714745527230478 us; error 58 us; skew 500 ppm
I20240503 14:12:07.230929 11710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.233386 11710 webserver.cc:460] Webserver started at http://127.10.145.62:40371/ using document root <none> and password file <none>
I20240503 14:12:07.234144 11710 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.234259 11710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.234596 11710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.237321 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "821c839181f648b19beed21bc3157ead"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v549"
I20240503 14:12:07.238101 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "821c839181f648b19beed21bc3157ead"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v549"
I20240503 14:12:07.243510 11710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:12:07.247212 11724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.248759 11710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:07.248925 11710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
uuid: "821c839181f648b19beed21bc3157ead"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v549"
I20240503 14:12:07.249126 11710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:07.275544 11710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:07.276679 11710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:07.276922 11710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:07.294442 11710 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.62:41805
I20240503 14:12:07.294467 11776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.62:41805 every 8 connection(s)
I20240503 14:12:07.295912 11710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:12:07.299674 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 11710
I20240503 14:12:07.299823 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.299860 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:12:07.313119 11777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead: Bootstrap starting.
I20240503 14:12:07.316402 11777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.317463 11777 log.cc:824] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:07.320109 11777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead: No bootstrap required, opened a new log
I20240503 14:12:07.326400 11777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821c839181f648b19beed21bc3157ead" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 41805 } }
I20240503 14:12:07.326756 11777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.326836 11777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821c839181f648b19beed21bc3157ead, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.327399 11777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821c839181f648b19beed21bc3157ead" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 41805 } }
I20240503 14:12:07.327592 11777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:07.327796 11777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:07.328015 11777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.330369 11777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821c839181f648b19beed21bc3157ead" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 41805 } }
I20240503 14:12:07.330986 11777 leader_election.cc:302] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 821c839181f648b19beed21bc3157ead; no voters:
I20240503 14:12:07.331719 11777 leader_election.cc:288] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:07.331888 11782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.332531 11782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [term 1 LEADER]: Becoming Leader. State: Replica: 821c839181f648b19beed21bc3157ead, State: Running, Role: LEADER
I20240503 14:12:07.333106 11782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821c839181f648b19beed21bc3157ead" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 41805 } }
I20240503 14:12:07.334149 11777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:07.336308 11784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader 821c839181f648b19beed21bc3157ead. Latest consensus state: current_term: 1 leader_uuid: "821c839181f648b19beed21bc3157ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821c839181f648b19beed21bc3157ead" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 41805 } } }
I20240503 14:12:07.336661 11784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:07.336282 11783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "821c839181f648b19beed21bc3157ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821c839181f648b19beed21bc3157ead" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 41805 } } }
I20240503 14:12:07.337437 11783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:07.340274 11788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:07.345803 11788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:07.355731 11788 catalog_manager.cc:1347] Generated new cluster ID: 506dd52d0cc4428c929770c7b6e8c20e
I20240503 14:12:07.355863 11788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:07.368788 11788 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:07.370209 11788 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:07.382386 11788 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 821c839181f648b19beed21bc3157ead: Generated new TSK 0
I20240503 14:12:07.383309 11788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:07.399266 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.1:0
--local_ip_for_outbound_sockets=127.10.145.1
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:41805
--builtin_ntp_servers=127.10.145.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:07.745608 11801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:07.745915 11801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:07.746017 11801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:07.753662 11801 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:07.753796 11801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:07.753926 11801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.1
I20240503 14:12:07.763123 11801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:41805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.1
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:07.764572 11801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:07.766676 11801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:07.779295 11809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.779676 11807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.779953 11806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.780536 11801 server_base.cc:1034] running on GCE node
I20240503 14:12:07.781162 11801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:07.782232 11801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:07.783437 11801 hybrid_clock.cc:648] HybridClock initialized: now 1714745527783394 us; error 31 us; skew 500 ppm
I20240503 14:12:07.783766 11801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.786073 11801 webserver.cc:460] Webserver started at http://127.10.145.1:41407/ using document root <none> and password file <none>
I20240503 14:12:07.786816 11801 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.786917 11801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.787246 11801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.789923 11801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "983f4b5044024998b3da233e01a32189"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v549"
I20240503 14:12:07.790663 11801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "983f4b5044024998b3da233e01a32189"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v549"
I20240503 14:12:07.795897 11801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:12:07.799253 11815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.800748 11801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:07.800930 11801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "983f4b5044024998b3da233e01a32189"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-v549"
I20240503 14:12:07.801157 11801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:07.817554 11801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:07.818532 11801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:07.818835 11801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:07.819877 11801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:07.821604 11801 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:07.821713 11801 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.821831 11801 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:07.821906 11801 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.854171 11801 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.1:38953
I20240503 14:12:07.854216 11928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.1:38953 every 8 connection(s)
I20240503 14:12:07.855623 11801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:12:07.860222 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 11801
I20240503 14:12:07.860451 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:12:07.864274 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.2:0
--local_ip_for_outbound_sockets=127.10.145.2
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:41805
--builtin_ntp_servers=127.10.145.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:07.870277 11929 heartbeater.cc:347] Connected to a master server at 127.10.145.62:41805
I20240503 14:12:07.870594 11929 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:07.871460 11929 heartbeater.cc:511] Master 127.10.145.62:41805 requested a full tablet report, sending...
I20240503 14:12:07.873615 11741 ts_manager.cc:194] Registered new tserver with Master: 983f4b5044024998b3da233e01a32189 (127.10.145.1:38953)
I20240503 14:12:07.875160 11741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.1:49903
W20240503 14:12:08.194759 11933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:08.195094 11933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:08.195235 11933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:08.203188 11933 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:08.203341 11933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:08.203500 11933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.2
I20240503 14:12:08.212857 11933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:41805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.2
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:08.214326 11933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:08.216176 11933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:08.229427 11939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.229596 11941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.229929 11938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:08.230484 11933 server_base.cc:1034] running on GCE node
I20240503 14:12:08.231225 11933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:08.232384 11933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:08.233593 11933 hybrid_clock.cc:648] HybridClock initialized: now 1714745528233523 us; error 62 us; skew 500 ppm
I20240503 14:12:08.233958 11933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:08.236459 11933 webserver.cc:460] Webserver started at http://127.10.145.2:38059/ using document root <none> and password file <none>
I20240503 14:12:08.237247 11933 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:08.237396 11933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:08.237732 11933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:08.240487 11933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c241312b45954a3db36a12ee894114e6"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-v549"
I20240503 14:12:08.241356 11933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c241312b45954a3db36a12ee894114e6"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-v549"
I20240503 14:12:08.246855 11933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20240503 14:12:08.250408 11947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.251843 11933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:12:08.251999 11933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c241312b45954a3db36a12ee894114e6"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-v549"
I20240503 14:12:08.252193 11933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:08.278786 11933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:08.279731 11933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:08.280017 11933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:08.281102 11933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:08.282739 11933 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:08.282825 11933 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.282927 11933 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:08.282997 11933 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.313884 11933 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.2:44827
I20240503 14:12:08.313952 12060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.2:44827 every 8 connection(s)
I20240503 14:12:08.315244 11933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:12:08.322790 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 11933
I20240503 14:12:08.322932 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:12:08.326457 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.3:0
--local_ip_for_outbound_sockets=127.10.145.3
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:41805
--builtin_ntp_servers=127.10.145.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:08.328258 12061 heartbeater.cc:347] Connected to a master server at 127.10.145.62:41805
I20240503 14:12:08.328578 12061 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.329557 12061 heartbeater.cc:511] Master 127.10.145.62:41805 requested a full tablet report, sending...
I20240503 14:12:08.331108 11741 ts_manager.cc:194] Registered new tserver with Master: c241312b45954a3db36a12ee894114e6 (127.10.145.2:44827)
I20240503 14:12:08.331934 11741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.2:43993
W20240503 14:12:08.655544 12064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:08.655903 12064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:08.656036 12064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:08.666260 12064 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:08.666414 12064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:08.666571 12064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.3
I20240503 14:12:08.676038 12064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:41805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.3
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:08.677476 12064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:08.679288 12064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:08.691737 12072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.692111 12070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.692453 12069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:08.692910 12064 server_base.cc:1034] running on GCE node
I20240503 14:12:08.693598 12064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:08.694617 12064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:08.695825 12064 hybrid_clock.cc:648] HybridClock initialized: now 1714745528695775 us; error 44 us; skew 500 ppm
I20240503 14:12:08.696158 12064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:08.698596 12064 webserver.cc:460] Webserver started at http://127.10.145.3:35747/ using document root <none> and password file <none>
I20240503 14:12:08.699362 12064 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:08.699478 12064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:08.699802 12064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:08.702525 12064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "73d0ccef96ea4b0e90f178c3eba7ff80"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-v549"
I20240503 14:12:08.703298 12064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "73d0ccef96ea4b0e90f178c3eba7ff80"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-v549"
I20240503 14:12:08.708737 12064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:12:08.712311 12078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.713853 12064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:08.714084 12064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "73d0ccef96ea4b0e90f178c3eba7ff80"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-v549"
I20240503 14:12:08.714308 12064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:08.750608 12064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:08.751533 12064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:08.751847 12064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:08.752858 12064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:08.754509 12064 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:08.754603 12064 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.754724 12064 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:08.754788 12064 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.784085 12064 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.3:34599
I20240503 14:12:08.784188 12190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.3:34599 every 8 connection(s)
I20240503 14:12:08.785454 12064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:08.786404 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 12064
I20240503 14:12:08.786592 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:12:08.790130 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.4:0
--local_ip_for_outbound_sockets=127.10.145.4
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:41805
--builtin_ntp_servers=127.10.145.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:08.803202 12191 heartbeater.cc:347] Connected to a master server at 127.10.145.62:41805
I20240503 14:12:08.803541 12191 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.804425 12191 heartbeater.cc:511] Master 127.10.145.62:41805 requested a full tablet report, sending...
I20240503 14:12:08.806035 11741 ts_manager.cc:194] Registered new tserver with Master: 73d0ccef96ea4b0e90f178c3eba7ff80 (127.10.145.3:34599)
I20240503 14:12:08.806892 11741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.3:44373
I20240503 14:12:08.878214 11929 heartbeater.cc:503] Master 127.10.145.62:41805 was elected leader, sending a full tablet report...
W20240503 14:12:09.114670 12194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:09.115001 12194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:09.115121 12194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:09.122761 12194 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:09.122923 12194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:09.123085 12194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.4
I20240503 14:12:09.132494 12194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:41805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.4
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:09.133980 12194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:09.135807 12194 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:09.148099 12199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.148504 12200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.148849 12202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:09.149315 12194 server_base.cc:1034] running on GCE node
I20240503 14:12:09.149947 12194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:09.150920 12194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:09.152149 12194 hybrid_clock.cc:648] HybridClock initialized: now 1714745529152094 us; error 47 us; skew 500 ppm
I20240503 14:12:09.152516 12194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:09.154969 12194 webserver.cc:460] Webserver started at http://127.10.145.4:46527/ using document root <none> and password file <none>
I20240503 14:12:09.155735 12194 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:09.155846 12194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:09.156193 12194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:09.159010 12194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "185f051d7d84476898a5005b08de4da5"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-v549"
I20240503 14:12:09.159819 12194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "185f051d7d84476898a5005b08de4da5"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-v549"
I20240503 14:12:09.165261 12194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 14:12:09.168995 12208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.170481 12194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:09.170648 12194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "185f051d7d84476898a5005b08de4da5"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-v549"
I20240503 14:12:09.170864 12194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:09.222029 12194 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:09.222945 12194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:09.223215 12194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:09.224241 12194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:09.225842 12194 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:09.225941 12194 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.226048 12194 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:09.226114 12194 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.256477 12194 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.4:45257
I20240503 14:12:09.256536 12320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.4:45257 every 8 connection(s)
I20240503 14:12:09.257934 12194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:09.267539 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 12194
I20240503 14:12:09.267760 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:12:09.270841 12321 heartbeater.cc:347] Connected to a master server at 127.10.145.62:41805
I20240503 14:12:09.271224 12321 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:09.271256 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.5:0
--local_ip_for_outbound_sockets=127.10.145.5
--webserver_interface=127.10.145.5
--webserver_port=0
--tserver_master_addrs=127.10.145.62:41805
--builtin_ntp_servers=127.10.145.20:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:09.272433 12321 heartbeater.cc:511] Master 127.10.145.62:41805 requested a full tablet report, sending...
I20240503 14:12:09.274199 11741 ts_manager.cc:194] Registered new tserver with Master: 185f051d7d84476898a5005b08de4da5 (127.10.145.4:45257)
I20240503 14:12:09.275022 11741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.4:39377
I20240503 14:12:09.335179 12061 heartbeater.cc:503] Master 127.10.145.62:41805 was elected leader, sending a full tablet report...
W20240503 14:12:09.607501 12324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:09.607807 12324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:09.607935 12324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:09.615548 12324 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:09.615710 12324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:09.615875 12324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.5
I20240503 14:12:09.625077 12324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.10.145.5
--webserver_port=0
--tserver_master_addrs=127.10.145.62:41805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.5
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:09.626499 12324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:09.628405 12324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:09.641144 12329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:09.641788 12324 server_base.cc:1034] running on GCE node
W20240503 14:12:09.642164 12332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.642184 12330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:09.642979 12324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:09.643950 12324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:09.645203 12324 hybrid_clock.cc:648] HybridClock initialized: now 1714745529645149 us; error 47 us; skew 500 ppm
I20240503 14:12:09.645609 12324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:09.648082 12324 webserver.cc:460] Webserver started at http://127.10.145.5:44153/ using document root <none> and password file <none>
I20240503 14:12:09.648833 12324 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:09.648931 12324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:09.649223 12324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:09.651961 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "043bebcd8867402499f7f6f52ca65dda"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-v549"
I20240503 14:12:09.652812 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "043bebcd8867402499f7f6f52ca65dda"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-v549"
I20240503 14:12:09.658350 12324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240503 14:12:09.661895 12338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.663475 12324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:12:09.663806 12324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "043bebcd8867402499f7f6f52ca65dda"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-v549"
I20240503 14:12:09.664278 12324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:09.681923 12324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:09.682812 12324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:09.683074 12324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:09.684038 12324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:09.685628 12324 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:09.685719 12324 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.685802 12324 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:09.685851 12324 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.715735 12324 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.5:41171
I20240503 14:12:09.715782 12450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.5:41171 every 8 connection(s)
I20240503 14:12:09.717048 12324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:09.719499 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 12324
I20240503 14:12:09.719679 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:12:09.729707 12451 heartbeater.cc:347] Connected to a master server at 127.10.145.62:41805
I20240503 14:12:09.730072 12451 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:09.730938 12451 heartbeater.cc:511] Master 127.10.145.62:41805 requested a full tablet report, sending...
I20240503 14:12:09.732461 11741 ts_manager.cc:194] Registered new tserver with Master: 043bebcd8867402499f7f6f52ca65dda (127.10.145.5:41171)
I20240503 14:12:09.733253 11741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.5:35229
I20240503 14:12:09.735795 10820 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:12:09.752979 11741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54066:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:12:09.788039 11995 tablet_service.cc:1467] Processing CreateTablet for tablet d65b921230304d91a703340e2ca92542 (DEFAULT_TABLE table=TestTable [id=9d1b0b4f73aa494a8add6e7487d5494d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.788482 12126 tablet_service.cc:1467] Processing CreateTablet for tablet d65b921230304d91a703340e2ca92542 (DEFAULT_TABLE table=TestTable [id=9d1b0b4f73aa494a8add6e7487d5494d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.788470 12256 tablet_service.cc:1467] Processing CreateTablet for tablet d65b921230304d91a703340e2ca92542 (DEFAULT_TABLE table=TestTable [id=9d1b0b4f73aa494a8add6e7487d5494d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.790007 11995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65b921230304d91a703340e2ca92542. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.790556 12256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65b921230304d91a703340e2ca92542. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.791453 12126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65b921230304d91a703340e2ca92542. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.803495 12469 tablet_bootstrap.cc:492] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5: Bootstrap starting.
I20240503 14:12:09.805842 12470 tablet_bootstrap.cc:492] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6: Bootstrap starting.
I20240503 14:12:09.806782 12471 tablet_bootstrap.cc:492] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80: Bootstrap starting.
I20240503 14:12:09.807657 12469 tablet_bootstrap.cc:654] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.809042 12469 log.cc:824] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:09.809510 12191 heartbeater.cc:503] Master 127.10.145.62:41805 was elected leader, sending a full tablet report...
I20240503 14:12:09.810720 12471 tablet_bootstrap.cc:654] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.811549 12470 tablet_bootstrap.cc:654] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.812167 12471 log.cc:824] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:09.812525 12470 log.cc:824] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:09.815311 12471 tablet_bootstrap.cc:492] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80: No bootstrap required, opened a new log
I20240503 14:12:09.815588 12471 ts_tablet_manager.cc:1367] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:12:09.816152 12469 tablet_bootstrap.cc:492] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5: No bootstrap required, opened a new log
I20240503 14:12:09.816442 12469 ts_tablet_manager.cc:1367] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20240503 14:12:09.816723 12470 tablet_bootstrap.cc:492] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6: No bootstrap required, opened a new log
I20240503 14:12:09.816994 12470 ts_tablet_manager.cc:1367] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20240503 14:12:09.822371 12471 raft_consensus.cc:348] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 } } peers { permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 } } peers { permanent_uuid: "185f051d7d84476898a5005b08de4da5" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45257 } }
I20240503 14:12:09.822760 12471 raft_consensus.cc:374] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.822878 12471 raft_consensus.cc:729] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d0ccef96ea4b0e90f178c3eba7ff80, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.823622 12471 consensus_queue.cc:260] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 } } peers { permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 } } peers { permanent_uuid: "185f051d7d84476898a5005b08de4da5" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45257 } }
I20240503 14:12:09.825356 12471 ts_tablet_manager.cc:1398] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20240503 14:12:09.825778 12469 raft_consensus.cc:348] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 } } peers { permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 } } peers { permanent_uuid: "185f051d7d84476898a5005b08de4da5" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45257 } }
I20240503 14:12:09.826251 12469 raft_consensus.cc:374] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.826350 12469 raft_consensus.cc:729] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185f051d7d84476898a5005b08de4da5, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.827194 12469 consensus_queue.cc:260] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 } } peers { permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 } } peers { permanent_uuid: "185f051d7d84476898a5005b08de4da5" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45257 } }
I20240503 14:12:09.828063 12470 raft_consensus.cc:348] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 } } peers { permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 } } peers { permanent_uuid: "185f051d7d84476898a5005b08de4da5" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45257 } }
I20240503 14:12:09.828534 12470 raft_consensus.cc:374] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.828635 12470 raft_consensus.cc:729] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c241312b45954a3db36a12ee894114e6, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.828696 12321 heartbeater.cc:503] Master 127.10.145.62:41805 was elected leader, sending a full tablet report...
I20240503 14:12:09.829445 12469 ts_tablet_manager.cc:1398] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20240503 14:12:09.829527 12470 consensus_queue.cc:260] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 } } peers { permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 } } peers { permanent_uuid: "185f051d7d84476898a5005b08de4da5" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45257 } }
I20240503 14:12:09.834872 12470 ts_tablet_manager.cc:1398] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6: Time spent starting tablet: real 0.018s user 0.010s sys 0.006s
I20240503 14:12:09.934679 12476 raft_consensus.cc:482] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.934979 12476 raft_consensus.cc:504] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 } } peers { permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 } } peers { permanent_uuid: "185f051d7d84476898a5005b08de4da5" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45257 } }
I20240503 14:12:09.936789 12476 leader_election.cc:288] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73d0ccef96ea4b0e90f178c3eba7ff80 (127.10.145.3:34599), c241312b45954a3db36a12ee894114e6 (127.10.145.2:44827)
I20240503 14:12:09.943917 12015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65b921230304d91a703340e2ca92542" candidate_uuid: "185f051d7d84476898a5005b08de4da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c241312b45954a3db36a12ee894114e6" is_pre_election: true
I20240503 14:12:09.943920 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65b921230304d91a703340e2ca92542" candidate_uuid: "185f051d7d84476898a5005b08de4da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" is_pre_election: true
I20240503 14:12:09.944406 12015 raft_consensus.cc:2454] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185f051d7d84476898a5005b08de4da5 in term 0.
I20240503 14:12:09.944422 12146 raft_consensus.cc:2454] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185f051d7d84476898a5005b08de4da5 in term 0.
I20240503 14:12:09.945029 12212 leader_election.cc:302] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185f051d7d84476898a5005b08de4da5, c241312b45954a3db36a12ee894114e6; no voters:
I20240503 14:12:09.945426 12476 raft_consensus.cc:2781] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.945557 12476 raft_consensus.cc:482] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.945663 12476 raft_consensus.cc:3037] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.947773 12476 raft_consensus.cc:504] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 } } peers { permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 } } peers { permanent_uuid: "185f051d7d84476898a5005b08de4da5" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45257 } }
I20240503 14:12:09.948462 12476 leader_election.cc:288] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [CANDIDATE]: Term 1 election: Requested vote from peers 73d0ccef96ea4b0e90f178c3eba7ff80 (127.10.145.3:34599), c241312b45954a3db36a12ee894114e6 (127.10.145.2:44827)
I20240503 14:12:09.948971 12146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65b921230304d91a703340e2ca92542" candidate_uuid: "185f051d7d84476898a5005b08de4da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80"
I20240503 14:12:09.949038 12015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65b921230304d91a703340e2ca92542" candidate_uuid: "185f051d7d84476898a5005b08de4da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c241312b45954a3db36a12ee894114e6"
I20240503 14:12:09.949203 12146 raft_consensus.cc:3037] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.949244 12015 raft_consensus.cc:3037] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.952353 12146 raft_consensus.cc:2454] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185f051d7d84476898a5005b08de4da5 in term 1.
I20240503 14:12:09.952353 12015 raft_consensus.cc:2454] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185f051d7d84476898a5005b08de4da5 in term 1.
I20240503 14:12:09.952837 12212 leader_election.cc:302] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185f051d7d84476898a5005b08de4da5, c241312b45954a3db36a12ee894114e6; no voters:
I20240503 14:12:09.953210 12476 raft_consensus.cc:2781] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.953649 12476 raft_consensus.cc:686] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [term 1 LEADER]: Becoming Leader. State: Replica: 185f051d7d84476898a5005b08de4da5, State: Running, Role: LEADER
I20240503 14:12:09.954216 12476 consensus_queue.cc:237] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 } } peers { permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 } } peers { permanent_uuid: "185f051d7d84476898a5005b08de4da5" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45257 } }
I20240503 14:12:09.959007 11741 catalog_manager.cc:5484] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 185f051d7d84476898a5005b08de4da5 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "185f051d7d84476898a5005b08de4da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "185f051d7d84476898a5005b08de4da5" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45257 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:09.979010 10820 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:12:09.981945 10820 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 983f4b5044024998b3da233e01a32189 to finish bootstrapping
I20240503 14:12:09.991755 10820 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c241312b45954a3db36a12ee894114e6 to finish bootstrapping
I20240503 14:12:10.000166 10820 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 73d0ccef96ea4b0e90f178c3eba7ff80 to finish bootstrapping
I20240503 14:12:10.008311 10820 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 185f051d7d84476898a5005b08de4da5 to finish bootstrapping
W20240503 14:12:10.009935 12322 tablet.cc:2369] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:10.016280 10820 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 043bebcd8867402499f7f6f52ca65dda to finish bootstrapping
W20240503 14:12:10.038093 12192 tablet.cc:2369] T d65b921230304d91a703340e2ca92542 P 73d0ccef96ea4b0e90f178c3eba7ff80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:10.068099 12062 tablet.cc:2369] T d65b921230304d91a703340e2ca92542 P c241312b45954a3db36a12ee894114e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:10.371439 12476 consensus_queue.cc:1035] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73d0ccef96ea4b0e90f178c3eba7ff80" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 34599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.380254 12476 consensus_queue.cc:1035] T d65b921230304d91a703340e2ca92542 P 185f051d7d84476898a5005b08de4da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c241312b45954a3db36a12ee894114e6" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 44827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.513749 12386 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.517580 12256 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:10.518352 12126 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.519855 11863 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.520921 11995 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.698709 12256 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:10.699335 12386 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.718075 11995 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.718741 11863 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.719053 12126 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.736186 12451 heartbeater.cc:503] Master 127.10.145.62:41805 was elected leader, sending a full tablet report...
I20240503 14:12:10.892232 12386 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.899672 12256 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:10.905349 12126 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.915715 11863 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.917222 11995 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.541988 12386 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.545917 12256 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:11.549389 11995 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.561398 11863 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.587746 12126 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.731410 12386 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.735757 12256 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:11.741787 11863 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.759636 12126 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.762100 11995 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.932076 12386 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.933902 12256 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:11.942054 12126 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.942378 11863 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.942611 11995 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:12.103752 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 11801
I20240503 14:12:12.105317 11924 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.199282 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 11801
I20240503 14:12:12.214089 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 11933
I20240503 14:12:12.215176 12056 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.312781 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 11933
I20240503 14:12:12.327418 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 12064
I20240503 14:12:12.328473 12186 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.427037 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 12064
W20240503 14:12:12.443035 11727 connection.cc:535] server connection from 127.10.145.3:44373 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:12.443105 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 12194
I20240503 14:12:12.444173 12316 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.543568 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 12194
I20240503 14:12:12.558373 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 12324
I20240503 14:12:12.559558 12446 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.652205 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 12324
I20240503 14:12:12.667781 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 11710
I20240503 14:12:12.668972 11772 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.786633 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 11710
2024-05-03T14:12:12Z chronyd exiting
[ OK ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 (5996 ms)
[----------] 1 test from RebalanceStartSafetyTest (5996 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/7
I20240503 14:12:12.826162 10820 test_util.cc:268] Using random seed: 1314734598
I20240503 14:12:12.828083 10820 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:12:12.828166 10820 ts_itest-base.cc:113] --------------
I20240503 14:12:12.828197 10820 ts_itest-base.cc:114] 7 tablet servers
I20240503 14:12:12.828256 10820 ts_itest-base.cc:115] 5 replicas per TS
I20240503 14:12:12.828298 10820 ts_itest-base.cc:116] --------------
2024-05-03T14:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:12:12Z Disabled control of system clock
I20240503 14:12:12.852298 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:35023
--webserver_interface=127.10.145.62
--webserver_port=0
--builtin_ntp_servers=127.10.145.20:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.145.62:35023
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 14:12:13.190855 12711 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 14:12:13.191200 12711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:13.191298 12711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:13.191399 12711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:13.199047 12711 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:13.199177 12711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:13.199244 12711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:13.199306 12711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:13.199369 12711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:13.208868 12711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.10.145.62:35023
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:35023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:13.210292 12711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:13.212211 12711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:13.223866 12716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.224218 12717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.224594 12719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:13.225106 12711 server_base.cc:1034] running on GCE node
I20240503 14:12:13.226052 12711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:13.227715 12711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:13.228987 12711 hybrid_clock.cc:648] HybridClock initialized: now 1714745533228910 us; error 55 us; skew 500 ppm
I20240503 14:12:13.229436 12711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:13.232270 12711 webserver.cc:460] Webserver started at http://127.10.145.62:45571/ using document root <none> and password file <none>
I20240503 14:12:13.233054 12711 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:13.233175 12711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:13.233531 12711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:13.236452 12711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "587e2eb33b9846ca8ab894d34f10c964"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-v549"
I20240503 14:12:13.237267 12711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "587e2eb33b9846ca8ab894d34f10c964"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-v549"
I20240503 14:12:13.243093 12711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240503 14:12:13.247154 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.248741 12711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:13.248924 12711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
uuid: "587e2eb33b9846ca8ab894d34f10c964"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-v549"
I20240503 14:12:13.249126 12711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:13.272012 12711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:13.272933 12711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:13.273226 12711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:13.291498 12711 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.62:35023
I20240503 14:12:13.291517 12776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.62:35023 every 8 connection(s)
I20240503 14:12:13.293002 12711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:12:13.297391 12777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.300506 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 12711
I20240503 14:12:13.300732 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:12:13.308351 12777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964: Bootstrap starting.
I20240503 14:12:13.311648 12777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.312892 12777 log.cc:824] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:13.315898 12777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964: No bootstrap required, opened a new log
I20240503 14:12:13.322751 12777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e2eb33b9846ca8ab894d34f10c964" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 35023 } }
I20240503 14:12:13.323168 12777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.323249 12777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587e2eb33b9846ca8ab894d34f10c964, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.323841 12777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e2eb33b9846ca8ab894d34f10c964" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 35023 } }
I20240503 14:12:13.324038 12777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:13.324136 12777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:13.324260 12777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.326515 12777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e2eb33b9846ca8ab894d34f10c964" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 35023 } }
I20240503 14:12:13.327064 12777 leader_election.cc:302] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 587e2eb33b9846ca8ab894d34f10c964; no voters:
I20240503 14:12:13.327688 12777 leader_election.cc:288] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:13.327873 12782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.328508 12782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [term 1 LEADER]: Becoming Leader. State: Replica: 587e2eb33b9846ca8ab894d34f10c964, State: Running, Role: LEADER
I20240503 14:12:13.329099 12782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e2eb33b9846ca8ab894d34f10c964" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 35023 } }
I20240503 14:12:13.329833 12777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:13.332347 12783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "587e2eb33b9846ca8ab894d34f10c964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e2eb33b9846ca8ab894d34f10c964" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 35023 } } }
I20240503 14:12:13.332635 12783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:13.332985 12784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 587e2eb33b9846ca8ab894d34f10c964. Latest consensus state: current_term: 1 leader_uuid: "587e2eb33b9846ca8ab894d34f10c964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587e2eb33b9846ca8ab894d34f10c964" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 35023 } } }
I20240503 14:12:13.333273 12784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:13.333525 12788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:13.348714 12788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:13.359325 12788 catalog_manager.cc:1347] Generated new cluster ID: 33f92e8c8402466b9eeb3953e9cecb75
I20240503 14:12:13.359467 12788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:13.372078 12788 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:13.373462 12788 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:13.383148 12788 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 587e2eb33b9846ca8ab894d34f10c964: Generated new TSK 0
I20240503 14:12:13.384099 12788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:13.392509 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.1:0
--local_ip_for_outbound_sockets=127.10.145.1
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--builtin_ntp_servers=127.10.145.20:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:13.737715 12801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:13.738042 12801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:13.738130 12801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:13.745900 12801 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:13.746042 12801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:13.746160 12801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.1
I20240503 14:12:13.755769 12801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.1
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:13.757177 12801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:13.759249 12801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:13.772434 12806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.772797 12807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.773190 12809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:13.773670 12801 server_base.cc:1034] running on GCE node
I20240503 14:12:13.774360 12801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:13.775445 12801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:13.776671 12801 hybrid_clock.cc:648] HybridClock initialized: now 1714745533776603 us; error 66 us; skew 500 ppm
I20240503 14:12:13.777033 12801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:13.779657 12801 webserver.cc:460] Webserver started at http://127.10.145.1:38995/ using document root <none> and password file <none>
I20240503 14:12:13.780464 12801 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:13.780586 12801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:13.780910 12801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:13.783763 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "72b07e9b53b9471989185277045db8f7"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-v549"
I20240503 14:12:13.784623 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "72b07e9b53b9471989185277045db8f7"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-v549"
I20240503 14:12:13.790320 12801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240503 14:12:13.794266 12815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.795854 12801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:13.796047 12801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "72b07e9b53b9471989185277045db8f7"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-v549"
I20240503 14:12:13.796285 12801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:13.818835 12801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:13.819749 12801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:13.820035 12801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:13.821028 12801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:13.822665 12801 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:13.822772 12801 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.822885 12801 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:13.822952 12801 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.854110 12801 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.1:38531
I20240503 14:12:13.854166 12927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.1:38531 every 8 connection(s)
I20240503 14:12:13.855484 12801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:12:13.861099 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 12801
I20240503 14:12:13.861342 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:12:13.864495 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.2:0
--local_ip_for_outbound_sockets=127.10.145.2
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--builtin_ntp_servers=127.10.145.20:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:13.869374 12928 heartbeater.cc:347] Connected to a master server at 127.10.145.62:35023
I20240503 14:12:13.869715 12928 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:13.870604 12928 heartbeater.cc:511] Master 127.10.145.62:35023 requested a full tablet report, sending...
I20240503 14:12:13.873152 12742 ts_manager.cc:194] Registered new tserver with Master: 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531)
I20240503 14:12:13.875368 12742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.1:57343
W20240503 14:12:14.214579 12932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:14.214895 12932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:14.214985 12932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:14.222676 12932 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:14.222810 12932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:14.222942 12932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.2
I20240503 14:12:14.232597 12932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.2
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:14.234023 12932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:14.236076 12932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:14.248842 12938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.249213 12937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.249660 12940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:14.250931 12932 server_base.cc:1034] running on GCE node
I20240503 14:12:14.251616 12932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:14.252681 12932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:14.253952 12932 hybrid_clock.cc:648] HybridClock initialized: now 1714745534253865 us; error 61 us; skew 500 ppm
I20240503 14:12:14.254411 12932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:14.257411 12932 webserver.cc:460] Webserver started at http://127.10.145.2:38191/ using document root <none> and password file <none>
I20240503 14:12:14.258245 12932 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:14.258361 12932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:14.258703 12932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:14.261632 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3678714622584fa4883e4bf07fcabd75"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v549"
I20240503 14:12:14.262509 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3678714622584fa4883e4bf07fcabd75"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v549"
I20240503 14:12:14.268369 12932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:12:14.272621 12946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.274330 12932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:14.274524 12932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3678714622584fa4883e4bf07fcabd75"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v549"
I20240503 14:12:14.274753 12932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:14.307250 12932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:14.308336 12932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:14.308683 12932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:14.309973 12932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:14.312011 12932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:14.312125 12932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.312258 12932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:14.312340 12932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.343303 12932 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.2:36441
I20240503 14:12:14.343355 13058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.2:36441 every 8 connection(s)
I20240503 14:12:14.344702 12932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:12:14.345063 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 12932
I20240503 14:12:14.345314 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:12:14.349433 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.3:0
--local_ip_for_outbound_sockets=127.10.145.3
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--builtin_ntp_servers=127.10.145.20:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:14.359315 13059 heartbeater.cc:347] Connected to a master server at 127.10.145.62:35023
I20240503 14:12:14.359655 13059 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:14.360680 13059 heartbeater.cc:511] Master 127.10.145.62:35023 requested a full tablet report, sending...
I20240503 14:12:14.362677 12742 ts_manager.cc:194] Registered new tserver with Master: 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
I20240503 14:12:14.363873 12742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.2:44169
W20240503 14:12:14.692647 13062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:14.692970 13062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:14.693059 13062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:14.700639 13062 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:14.700779 13062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:14.700906 13062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.3
I20240503 14:12:14.710350 13062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.3
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:14.711853 13062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:14.714408 13062 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:14.727845 13070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.728149 13067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.728515 13068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:14.729058 13062 server_base.cc:1034] running on GCE node
I20240503 14:12:14.729847 13062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:14.730979 13062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:14.732219 13062 hybrid_clock.cc:648] HybridClock initialized: now 1714745534732165 us; error 45 us; skew 500 ppm
I20240503 14:12:14.732589 13062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:14.735245 13062 webserver.cc:460] Webserver started at http://127.10.145.3:35525/ using document root <none> and password file <none>
I20240503 14:12:14.736085 13062 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:14.736207 13062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:14.736518 13062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:14.739519 13062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7b75d8c9e35a416ba24798fde8b9def0"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v549"
I20240503 14:12:14.740382 13062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7b75d8c9e35a416ba24798fde8b9def0"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v549"
I20240503 14:12:14.746420 13062 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240503 14:12:14.750515 13076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.752092 13062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20240503 14:12:14.752261 13062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7b75d8c9e35a416ba24798fde8b9def0"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-v549"
I20240503 14:12:14.752475 13062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:14.776613 13062 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:14.777585 13062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:14.777855 13062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:14.778955 13062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:14.780612 13062 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:14.780704 13062 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.780819 13062 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:14.780885 13062 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.812321 13062 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.3:41101
I20240503 14:12:14.812408 13188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.3:41101 every 8 connection(s)
I20240503 14:12:14.814129 13062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:14.818610 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 13062
I20240503 14:12:14.818804 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:12:14.822288 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.4:0
--local_ip_for_outbound_sockets=127.10.145.4
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--builtin_ntp_servers=127.10.145.20:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:14.829409 13189 heartbeater.cc:347] Connected to a master server at 127.10.145.62:35023
I20240503 14:12:14.829787 13189 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:14.830787 13189 heartbeater.cc:511] Master 127.10.145.62:35023 requested a full tablet report, sending...
I20240503 14:12:14.832708 12742 ts_manager.cc:194] Registered new tserver with Master: 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
I20240503 14:12:14.833896 12742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.3:36755
I20240503 14:12:14.879030 12928 heartbeater.cc:503] Master 127.10.145.62:35023 was elected leader, sending a full tablet report...
W20240503 14:12:15.166275 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:15.166591 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:15.166678 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:15.176548 13192 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:15.176730 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:15.176911 13192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.4
I20240503 14:12:15.187357 13192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.4
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:15.188768 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:15.190757 13192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:15.204566 13197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:15.204913 13198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:15.205401 13200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:15.206055 13192 server_base.cc:1034] running on GCE node
I20240503 14:12:15.206749 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:15.207813 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:15.209036 13192 hybrid_clock.cc:648] HybridClock initialized: now 1714745535208980 us; error 58 us; skew 500 ppm
I20240503 14:12:15.209447 13192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:15.212044 13192 webserver.cc:460] Webserver started at http://127.10.145.4:34537/ using document root <none> and password file <none>
I20240503 14:12:15.212859 13192 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:15.212977 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:15.213346 13192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:15.216166 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2cd2acf7d8274b5a95482dffd0823a08"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-v549"
I20240503 14:12:15.217062 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2cd2acf7d8274b5a95482dffd0823a08"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-v549"
I20240503 14:12:15.222975 13192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 14:12:15.226945 13206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.228585 13192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:15.228783 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2cd2acf7d8274b5a95482dffd0823a08"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-v549"
I20240503 14:12:15.229012 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:15.250272 13192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:15.251238 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:15.251545 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:15.252622 13192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:15.254479 13192 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:15.254588 13192 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.254710 13192 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:15.254794 13192 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.285820 13192 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.4:42597
I20240503 14:12:15.285867 13318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.4:42597 every 8 connection(s)
I20240503 14:12:15.287240 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:15.293489 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 13192
I20240503 14:12:15.293723 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:12:15.296932 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.5:0
--local_ip_for_outbound_sockets=127.10.145.5
--webserver_interface=127.10.145.5
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--builtin_ntp_servers=127.10.145.20:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:15.304219 13319 heartbeater.cc:347] Connected to a master server at 127.10.145.62:35023
I20240503 14:12:15.304571 13319 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:15.305544 13319 heartbeater.cc:511] Master 127.10.145.62:35023 requested a full tablet report, sending...
I20240503 14:12:15.307148 12742 ts_manager.cc:194] Registered new tserver with Master: 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:12:15.308022 12742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.4:43541
I20240503 14:12:15.367136 13059 heartbeater.cc:503] Master 127.10.145.62:35023 was elected leader, sending a full tablet report...
W20240503 14:12:15.655608 13322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:15.655933 13322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:15.656065 13322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:15.664296 13322 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:15.664471 13322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:15.664623 13322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.5
I20240503 14:12:15.674325 13322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.10.145.5
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.5
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:15.675976 13322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:15.678084 13322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:15.692858 13330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:15.693497 13328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:15.693907 13327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:15.694568 13322 server_base.cc:1034] running on GCE node
I20240503 14:12:15.695395 13322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:15.696712 13322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:15.697976 13322 hybrid_clock.cc:648] HybridClock initialized: now 1714745535697920 us; error 63 us; skew 500 ppm
I20240503 14:12:15.698371 13322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:15.700937 13322 webserver.cc:460] Webserver started at http://127.10.145.5:37043/ using document root <none> and password file <none>
I20240503 14:12:15.701891 13322 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:15.702036 13322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:15.702473 13322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:15.705430 13322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e148d8a3757340ab9288ea98cabb5ad3"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-v549"
I20240503 14:12:15.706323 13322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e148d8a3757340ab9288ea98cabb5ad3"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-v549"
I20240503 14:12:15.712209 13322 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 14:12:15.716619 13336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.718472 13322 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20240503 14:12:15.718669 13322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e148d8a3757340ab9288ea98cabb5ad3"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-v549"
I20240503 14:12:15.718909 13322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:15.737430 13322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:15.738380 13322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:15.738663 13322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:15.739692 13322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:15.741351 13322 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:15.741447 13322 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.741572 13322 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:15.741652 13322 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.772073 13322 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.5:38005
I20240503 14:12:15.772110 13448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.5:38005 every 8 connection(s)
I20240503 14:12:15.773651 13322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:15.777509 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 13322
I20240503 14:12:15.777696 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:12:15.781570 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.6:0
--local_ip_for_outbound_sockets=127.10.145.6
--webserver_interface=127.10.145.6
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--builtin_ntp_servers=127.10.145.20:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:15.788345 13449 heartbeater.cc:347] Connected to a master server at 127.10.145.62:35023
I20240503 14:12:15.788753 13449 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:15.789853 13449 heartbeater.cc:511] Master 127.10.145.62:35023 requested a full tablet report, sending...
I20240503 14:12:15.791651 12742 ts_manager.cc:194] Registered new tserver with Master: e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:15.792773 12742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.5:49201
I20240503 14:12:15.836839 13189 heartbeater.cc:503] Master 127.10.145.62:35023 was elected leader, sending a full tablet report...
W20240503 14:12:16.125718 13452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:16.126052 13452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:16.126191 13452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:16.134369 13452 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:16.134544 13452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:16.134742 13452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.6
I20240503 14:12:16.144199 13452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.10.145.6
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.6
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:16.145712 13452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:16.147640 13452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:16.162362 13458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:16.162680 13457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:16.163559 13460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:16.164299 13452 server_base.cc:1034] running on GCE node
I20240503 14:12:16.165004 13452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:16.166163 13452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:16.167418 13452 hybrid_clock.cc:648] HybridClock initialized: now 1714745536167372 us; error 42 us; skew 500 ppm
I20240503 14:12:16.167819 13452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:16.171252 13452 webserver.cc:460] Webserver started at http://127.10.145.6:32839/ using document root <none> and password file <none>
I20240503 14:12:16.172230 13452 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:16.172389 13452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:16.172809 13452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:16.176595 13452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "af88d71da3a54f779cf4f651f4393bb6"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-v549"
I20240503 14:12:16.177809 13452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "af88d71da3a54f779cf4f651f4393bb6"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-v549"
I20240503 14:12:16.185021 13452 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240503 14:12:16.189323 13466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.191076 13452 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:16.191256 13452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "af88d71da3a54f779cf4f651f4393bb6"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-v549"
I20240503 14:12:16.191470 13452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:16.206199 13452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:16.207226 13452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:16.207537 13452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:16.208626 13452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:16.210657 13452 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:16.210779 13452 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.210884 13452 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:16.210961 13452 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.243522 13452 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.6:36377
I20240503 14:12:16.243556 13578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.6:36377 every 8 connection(s)
I20240503 14:12:16.244980 13452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:12:16.251695 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 13452
I20240503 14:12:16.251948 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:12:16.255960 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.7:0
--local_ip_for_outbound_sockets=127.10.145.7
--webserver_interface=127.10.145.7
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--builtin_ntp_servers=127.10.145.20:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:16.260087 13579 heartbeater.cc:347] Connected to a master server at 127.10.145.62:35023
I20240503 14:12:16.260438 13579 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:16.261642 13579 heartbeater.cc:511] Master 127.10.145.62:35023 requested a full tablet report, sending...
I20240503 14:12:16.263306 12742 ts_manager.cc:194] Registered new tserver with Master: af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:16.264127 12742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.6:45741
I20240503 14:12:16.310961 13319 heartbeater.cc:503] Master 127.10.145.62:35023 was elected leader, sending a full tablet report...
W20240503 14:12:16.602375 13582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:16.602686 13582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:16.602815 13582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:16.610558 13582 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:16.610728 13582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:16.610890 13582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.7
I20240503 14:12:16.620620 13582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.10.145.7
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.7
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:16.622092 13582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:16.624089 13582 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:16.637147 13587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:16.637615 13590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:16.638025 13588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:16.638484 13582 server_base.cc:1034] running on GCE node
I20240503 14:12:16.639102 13582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:16.640128 13582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:16.641383 13582 hybrid_clock.cc:648] HybridClock initialized: now 1714745536641323 us; error 51 us; skew 500 ppm
I20240503 14:12:16.641778 13582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:16.644364 13582 webserver.cc:460] Webserver started at http://127.10.145.7:34589/ using document root <none> and password file <none>
I20240503 14:12:16.645120 13582 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:16.645241 13582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:16.645584 13582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:16.648298 13582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "21d8b64a50ba4c919cb9e711d18fb7de"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-v549"
I20240503 14:12:16.649103 13582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "21d8b64a50ba4c919cb9e711d18fb7de"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-v549"
I20240503 14:12:16.654933 13582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240503 14:12:16.659135 13596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.660727 13582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:16.660915 13582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "21d8b64a50ba4c919cb9e711d18fb7de"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-v549"
I20240503 14:12:16.661131 13582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:16.682534 13582 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:16.683480 13582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:16.683769 13582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:16.684805 13582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:16.686444 13582 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:16.686543 13582 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.686654 13582 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:16.686724 13582 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.717168 13582 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.7:44891
I20240503 14:12:16.717218 13708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.7:44891 every 8 connection(s)
I20240503 14:12:16.718660 13582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:12:16.725972 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 13582
I20240503 14:12:16.726150 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:12:16.732529 13709 heartbeater.cc:347] Connected to a master server at 127.10.145.62:35023
I20240503 14:12:16.732844 13709 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:16.733741 13709 heartbeater.cc:511] Master 127.10.145.62:35023 requested a full tablet report, sending...
I20240503 14:12:16.735339 12742 ts_manager.cc:194] Registered new tserver with Master: 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
I20240503 14:12:16.736222 12742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.7:60949
I20240503 14:12:16.742224 10820 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 14:12:16.759248 12742 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40090:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:12:16.796555 13449 heartbeater.cc:503] Master 127.10.145.62:35023 was elected leader, sending a full tablet report...
I20240503 14:12:16.813452 12994 tablet_service.cc:1467] Processing CreateTablet for tablet e82229abb5be4ed1abc3fa200671416e (DEFAULT_TABLE table=TestTable [id=ed6a4b9d46264728bfc80066eea83360]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.821635 12994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82229abb5be4ed1abc3fa200671416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.822288 13514 tablet_service.cc:1467] Processing CreateTablet for tablet e82229abb5be4ed1abc3fa200671416e (DEFAULT_TABLE table=TestTable [id=ed6a4b9d46264728bfc80066eea83360]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.822247 13384 tablet_service.cc:1467] Processing CreateTablet for tablet e82229abb5be4ed1abc3fa200671416e (DEFAULT_TABLE table=TestTable [id=ed6a4b9d46264728bfc80066eea83360]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.822811 13254 tablet_service.cc:1467] Processing CreateTablet for tablet e82229abb5be4ed1abc3fa200671416e (DEFAULT_TABLE table=TestTable [id=ed6a4b9d46264728bfc80066eea83360]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.823813 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82229abb5be4ed1abc3fa200671416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.823829 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82229abb5be4ed1abc3fa200671416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.824594 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82229abb5be4ed1abc3fa200671416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.843820 13730 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:16.848785 13731 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:16.850080 13732 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:16.850344 13730 tablet_bootstrap.cc:654] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.851761 13730 log.cc:824] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:16.854673 13732 tablet_bootstrap.cc:654] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.855348 13644 tablet_service.cc:1467] Processing CreateTablet for tablet e82229abb5be4ed1abc3fa200671416e (DEFAULT_TABLE table=TestTable [id=ed6a4b9d46264728bfc80066eea83360]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.856132 13732 log.cc:824] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:16.856459 13733 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:16.857872 13644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82229abb5be4ed1abc3fa200671416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.859256 13731 tablet_bootstrap.cc:654] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.862901 13733 tablet_bootstrap.cc:654] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.864455 13733 log.cc:824] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:16.867729 13731 log.cc:824] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:16.871560 13730 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:16.872046 13730 ts_tablet_manager.cc:1367] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.008s
I20240503 14:12:16.877375 13738 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
I20240503 14:12:16.878387 13732 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:16.878988 13732 ts_tablet_manager.cc:1367] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.011s
I20240503 14:12:16.879042 13733 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:16.879333 13733 ts_tablet_manager.cc:1367] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20240503 14:12:16.881815 13730 raft_consensus.cc:348] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.882477 13730 raft_consensus.cc:374] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.882648 13730 raft_consensus.cc:729] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.882700 13731 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:16.882994 13731 ts_tablet_manager.cc:1367] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.007s
I20240503 14:12:16.883981 13738 tablet_bootstrap.cc:654] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.885470 13738 log.cc:824] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:16.885447 13730 consensus_queue.cc:260] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.887898 13732 raft_consensus.cc:348] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.888525 13732 raft_consensus.cc:374] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.888684 13732 raft_consensus.cc:729] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.889699 13732 consensus_queue.cc:260] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.893617 13732 ts_tablet_manager.cc:1398] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20240503 14:12:16.898658 13731 raft_consensus.cc:348] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.899370 13731 raft_consensus.cc:374] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.899525 13731 raft_consensus.cc:729] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.901176 13731 consensus_queue.cc:260] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.903244 13730 ts_tablet_manager.cc:1398] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.031s user 0.024s sys 0.003s
I20240503 14:12:16.903441 13733 raft_consensus.cc:348] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.903991 13733 raft_consensus.cc:374] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.904152 13733 raft_consensus.cc:729] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.904870 13738 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: No bootstrap required, opened a new log
I20240503 14:12:16.905071 13733 consensus_queue.cc:260] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.905450 13738 ts_tablet_manager.cc:1367] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.009s
I20240503 14:12:16.910370 13733 ts_tablet_manager.cc:1398] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.021s user 0.014s sys 0.000s
I20240503 14:12:16.911557 13579 heartbeater.cc:503] Master 127.10.145.62:35023 was elected leader, sending a full tablet report...
I20240503 14:12:16.912434 13731 ts_tablet_manager.cc:1398] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.029s user 0.018s sys 0.000s
I20240503 14:12:16.916646 13738 raft_consensus.cc:348] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.917160 13738 raft_consensus.cc:374] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.917246 13738 raft_consensus.cc:729] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.917904 13738 consensus_queue.cc:260] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.918926 13709 heartbeater.cc:503] Master 127.10.145.62:35023 was elected leader, sending a full tablet report...
I20240503 14:12:16.919392 13738 ts_tablet_manager.cc:1398] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
I20240503 14:12:16.961802 13740 raft_consensus.cc:482] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:16.962075 13740 raft_consensus.cc:504] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.964172 13740 leader_election.cc:288] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
W20240503 14:12:16.970721 13710 tablet.cc:2369] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:16.974390 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:16.975046 13404 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:16.975518 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:16.976094 13534 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:16.976475 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" is_pre_election: true
I20240503 14:12:16.976704 12950 leader_election.cc:302] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:16.976963 13664 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:16.976892 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:16.977228 13740 raft_consensus.cc:2781] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:16.977419 13740 raft_consensus.cc:482] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:16.977424 13274 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:16.977522 13740 raft_consensus.cc:3037] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.979825 13740 raft_consensus.cc:504] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.980639 13740 leader_election.cc:288] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 election: Requested vote from peers 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
I20240503 14:12:16.981091 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:16.981130 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:16.981274 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:16.981333 13664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d8b64a50ba4c919cb9e711d18fb7de"
I20240503 14:12:16.981357 13274 raft_consensus.cc:3037] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.981382 13534 raft_consensus.cc:3037] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.981510 13404 raft_consensus.cc:3037] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.981534 13664 raft_consensus.cc:3037] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.984560 13664 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:16.984560 13274 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:16.984551 13404 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:16.984558 13534 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:16.985148 12948 leader_election.cc:302] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6; no voters:
I20240503 14:12:16.985520 13740 raft_consensus.cc:2781] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.986014 13740 raft_consensus.cc:686] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Becoming Leader. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Running, Role: LEADER
I20240503 14:12:16.986640 13740 consensus_queue.cc:237] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:16.991998 12742 catalog_manager.cc:5484] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3678714622584fa4883e4bf07fcabd75 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:16.997109 13580 tablet.cc:2369] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:17.026273 13450 tablet.cc:2369] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:17.040239 13320 tablet.cc:2369] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:17.048656 10820 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 14:12:17.051872 10820 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 72b07e9b53b9471989185277045db8f7 to finish bootstrapping
I20240503 14:12:17.064213 10820 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3678714622584fa4883e4bf07fcabd75 to finish bootstrapping
I20240503 14:12:17.074616 10820 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7b75d8c9e35a416ba24798fde8b9def0 to finish bootstrapping
I20240503 14:12:17.087249 10820 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2cd2acf7d8274b5a95482dffd0823a08 to finish bootstrapping
I20240503 14:12:17.095913 10820 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e148d8a3757340ab9288ea98cabb5ad3 to finish bootstrapping
W20240503 14:12:17.098559 13060 tablet.cc:2369] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:17.103924 10820 ts_itest-base.cc:233] Waiting for 1 tablets on tserver af88d71da3a54f779cf4f651f4393bb6 to finish bootstrapping
I20240503 14:12:17.111930 10820 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 21d8b64a50ba4c919cb9e711d18fb7de to finish bootstrapping
I20240503 14:12:17.120095 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 13582
I20240503 14:12:17.121179 13704 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:17.219329 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 13582
I20240503 14:12:17.398380 13740 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:17.542071 13740 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:17.550509 12948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.7:44891: connect: Connection refused (error 111)
I20240503 14:12:17.552083 13758 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:17.554208 12948 consensus_peers.cc:485] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.7:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:20.369959 12948 consensus_peers.cc:485] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.7:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:20.984591 12741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40090:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:20.997231 12994 tablet_service.cc:1467] Processing CreateTablet for tablet c43a1cb5209c4a69a95877c842460adc (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.997236 13254 tablet_service.cc:1467] Processing CreateTablet for tablet c43a1cb5209c4a69a95877c842460adc (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:20.997764 12994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43a1cb5209c4a69a95877c842460adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:20.997773 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43a1cb5209c4a69a95877c842460adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.006099 13384 tablet_service.cc:1467] Processing CreateTablet for tablet c43a1cb5209c4a69a95877c842460adc (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.006678 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43a1cb5209c4a69a95877c842460adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.007694 13786 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:21.009590 13786 tablet_bootstrap.cc:654] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.010084 13514 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfebd34071946398632ab7597622a41 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.010545 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfebd34071946398632ab7597622a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.010191 12863 tablet_service.cc:1467] Processing CreateTablet for tablet c43a1cb5209c4a69a95877c842460adc (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.011994 12863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43a1cb5209c4a69a95877c842460adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.017057 13787 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:21.019006 12862 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfebd34071946398632ab7597622a41 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.019040 13253 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfebd34071946398632ab7597622a41 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.019580 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfebd34071946398632ab7597622a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.019608 12862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfebd34071946398632ab7597622a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.030638 13787 tablet_bootstrap.cc:654] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.030813 13513 tablet_service.cc:1467] Processing CreateTablet for tablet 37542e9a30324860bb575ae02b3c92b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.030818 13383 tablet_service.cc:1467] Processing CreateTablet for tablet 37542e9a30324860bb575ae02b3c92b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.030467 13122 tablet_service.cc:1467] Processing CreateTablet for tablet 37542e9a30324860bb575ae02b3c92b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.031270 13513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37542e9a30324860bb575ae02b3c92b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.031280 13383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37542e9a30324860bb575ae02b3c92b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.032162 13122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37542e9a30324860bb575ae02b3c92b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.033362 13123 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfebd34071946398632ab7597622a41 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.033910 13123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfebd34071946398632ab7597622a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.034567 13124 tablet_service.cc:1467] Processing CreateTablet for tablet c43a1cb5209c4a69a95877c842460adc (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.036302 12863 tablet_service.cc:1467] Processing CreateTablet for tablet 37542e9a30324860bb575ae02b3c92b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.037251 12863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37542e9a30324860bb575ae02b3c92b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.040501 12993 tablet_service.cc:1467] Processing CreateTablet for tablet 37542e9a30324860bb575ae02b3c92b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.041039 12993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37542e9a30324860bb575ae02b3c92b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.035179 13124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43a1cb5209c4a69a95877c842460adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.049067 13792 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:21.050133 13787 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:21.050367 13787 ts_tablet_manager.cc:1367] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.014s
I20240503 14:12:21.050923 13792 tablet_bootstrap.cc:654] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.051474 13787 raft_consensus.cc:348] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.051826 13787 raft_consensus.cc:374] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.051985 13787 raft_consensus.cc:729] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.052306 13787 consensus_queue.cc:260] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.053063 13787 ts_tablet_manager.cc:1398] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:12:21.053092 13789 tablet_bootstrap.cc:492] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:21.057734 12994 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfebd34071946398632ab7597622a41 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.058162 12994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfebd34071946398632ab7597622a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.064524 13789 tablet_bootstrap.cc:654] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.065474 13787 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:21.065641 13794 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:21.066227 13790 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:21.067081 13786 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:21.067145 13787 tablet_bootstrap.cc:654] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.067257 13786 ts_tablet_manager.cc:1367] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20240503 14:12:21.068266 13786 raft_consensus.cc:348] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.068533 13786 raft_consensus.cc:374] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.068627 13786 raft_consensus.cc:729] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.068885 13786 consensus_queue.cc:260] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.069499 13790 tablet_bootstrap.cc:654] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.070667 13786 ts_tablet_manager.cc:1398] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:21.070160 13794 tablet_bootstrap.cc:654] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.070976 13786 tablet_bootstrap.cc:492] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:21.071038 13787 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:21.071198 13787 ts_tablet_manager.cc:1367] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240503 14:12:21.071981 13787 raft_consensus.cc:348] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.072248 13787 raft_consensus.cc:374] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.072338 13787 raft_consensus.cc:729] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.072530 13786 tablet_bootstrap.cc:654] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.072602 13787 consensus_queue.cc:260] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.073505 13789 log.cc:824] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:21.073661 13794 log.cc:824] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:21.074827 13792 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:21.074997 13792 ts_tablet_manager.cc:1367] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.003s
I20240503 14:12:21.075875 13787 ts_tablet_manager.cc:1398] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 14:12:21.075996 13792 raft_consensus.cc:348] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.076246 13792 raft_consensus.cc:374] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.076344 13792 raft_consensus.cc:729] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.076607 13792 consensus_queue.cc:260] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.078505 13787 tablet_bootstrap.cc:492] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:21.080456 13787 tablet_bootstrap.cc:654] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.084261 13789 tablet_bootstrap.cc:492] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:21.084596 13789 ts_tablet_manager.cc:1367] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.000s
I20240503 14:12:21.085526 13787 tablet_bootstrap.cc:492] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:21.085685 13787 ts_tablet_manager.cc:1367] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240503 14:12:21.086494 13787 raft_consensus.cc:348] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.086789 13787 raft_consensus.cc:374] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.086875 13787 raft_consensus.cc:729] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.087208 13786 tablet_bootstrap.cc:492] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:21.087203 13787 consensus_queue.cc:260] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.087424 13786 ts_tablet_manager.cc:1367] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20240503 14:12:21.088358 13794 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:21.088207 13786 raft_consensus.cc:348] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.088464 13786 raft_consensus.cc:374] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.088542 13786 raft_consensus.cc:729] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.088662 13794 ts_tablet_manager.cc:1367] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.007s
I20240503 14:12:21.088855 13786 consensus_queue.cc:260] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.089433 13786 ts_tablet_manager.cc:1398] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:12:21.093274 13787 ts_tablet_manager.cc:1398] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:12:21.096944 13792 ts_tablet_manager.cc:1398] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.022s user 0.001s sys 0.007s
I20240503 14:12:21.097358 13792 tablet_bootstrap.cc:492] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:21.097467 13794 raft_consensus.cc:348] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.097994 13794 raft_consensus.cc:374] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.098448 13794 raft_consensus.cc:729] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.099050 13792 tablet_bootstrap.cc:654] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.099356 13794 consensus_queue.cc:260] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.099725 13790 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:21.099947 13790 ts_tablet_manager.cc:1367] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.021s
I20240503 14:12:21.101090 13790 raft_consensus.cc:348] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.101513 13790 raft_consensus.cc:374] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.101649 13790 raft_consensus.cc:729] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.102022 13790 consensus_queue.cc:260] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.102811 13789 raft_consensus.cc:348] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.103355 13789 raft_consensus.cc:374] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.103461 13789 raft_consensus.cc:729] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.104386 13789 consensus_queue.cc:260] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.107842 13794 ts_tablet_manager.cc:1398] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
I20240503 14:12:21.108326 13794 tablet_bootstrap.cc:492] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:21.110132 13794 tablet_bootstrap.cc:654] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.112262 13792 tablet_bootstrap.cc:492] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:21.112468 13792 ts_tablet_manager.cc:1367] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20240503 14:12:21.113003 13790 ts_tablet_manager.cc:1398] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20240503 14:12:21.113361 13790 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:21.113319 13792 raft_consensus.cc:348] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.113605 13792 raft_consensus.cc:374] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.113695 13792 raft_consensus.cc:729] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.113984 13792 consensus_queue.cc:260] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.114661 13792 ts_tablet_manager.cc:1398] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:21.114818 13789 ts_tablet_manager.cc:1398] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.030s user 0.013s sys 0.007s
I20240503 14:12:21.115273 13789 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:21.115514 13790 tablet_bootstrap.cc:654] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.116948 13789 tablet_bootstrap.cc:654] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.120014 13758 raft_consensus.cc:482] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.120244 13758 raft_consensus.cc:504] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.121404 13794 tablet_bootstrap.cc:492] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:21.121592 13794 ts_tablet_manager.cc:1367] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 14:12:21.121570 13758 leader_election.cc:288] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:21.122618 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37542e9a30324860bb575ae02b3c92b9" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:21.122735 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37542e9a30324860bb575ae02b3c92b9" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:21.122684 13794 raft_consensus.cc:348] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.122875 13534 raft_consensus.cc:2454] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:21.122917 13794 raft_consensus.cc:374] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.123060 13794 raft_consensus.cc:729] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.123473 13794 consensus_queue.cc:260] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.124239 13794 ts_tablet_manager.cc:1398] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20240503 14:12:21.124655 12948 leader_election.cc:341] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:21.124747 13794 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:21.125196 13790 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:21.125458 13790 ts_tablet_manager.cc:1367] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20240503 14:12:21.126400 13790 raft_consensus.cc:348] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.126590 13794 tablet_bootstrap.cc:654] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.126713 13790 raft_consensus.cc:374] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.126809 13790 raft_consensus.cc:729] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.127101 13790 consensus_queue.cc:260] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.127784 13790 ts_tablet_manager.cc:1398] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:12:21.130005 13789 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:21.130257 13789 ts_tablet_manager.cc:1367] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20240503 14:12:21.131502 13789 raft_consensus.cc:348] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.131778 13789 raft_consensus.cc:374] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.131904 13789 raft_consensus.cc:729] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.132184 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37542e9a30324860bb575ae02b3c92b9" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:21.132239 13789 consensus_queue.cc:260] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.132931 13789 ts_tablet_manager.cc:1398] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:21.133333 13789 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
W20240503 14:12:21.133522 12947 leader_election.cc:341] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:21.135911 13789 tablet_bootstrap.cc:654] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.137492 13794 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:21.137681 13794 ts_tablet_manager.cc:1367] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240503 14:12:21.138068 13789 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:21.138249 13789 ts_tablet_manager.cc:1367] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:21.138520 13794 raft_consensus.cc:348] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.138828 13794 raft_consensus.cc:374] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.138922 13794 raft_consensus.cc:729] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.138916 13789 raft_consensus.cc:348] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.139174 13789 raft_consensus.cc:374] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.139254 13789 raft_consensus.cc:729] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.139233 13794 consensus_queue.cc:260] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.139566 13789 consensus_queue.cc:260] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.139890 13794 ts_tablet_manager.cc:1398] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:21.140100 13796 raft_consensus.cc:482] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.140167 13789 ts_tablet_manager.cc:1398] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:21.140239 13796 raft_consensus.cc:504] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.141799 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37542e9a30324860bb575ae02b3c92b9" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:21.142414 13144 raft_consensus.cc:2454] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:21.142657 13796 leader_election.cc:288] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:21.143828 12949 leader_election.cc:302] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0, af88d71da3a54f779cf4f651f4393bb6; no voters: 72b07e9b53b9471989185277045db8f7, e148d8a3757340ab9288ea98cabb5ad3
I20240503 14:12:21.144487 13758 raft_consensus.cc:2781] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.144593 13758 raft_consensus.cc:482] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.144686 13758 raft_consensus.cc:3037] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.147532 13758 raft_consensus.cc:504] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.149191 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37542e9a30324860bb575ae02b3c92b9" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:21.149518 12883 raft_consensus.cc:3037] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.152540 12883 raft_consensus.cc:2454] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:21.153789 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37542e9a30324860bb575ae02b3c92b9" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
I20240503 14:12:21.154033 13144 raft_consensus.cc:3037] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.155541 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37542e9a30324860bb575ae02b3c92b9" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:21.155558 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37542e9a30324860bb575ae02b3c92b9" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:21.155754 13404 raft_consensus.cc:3037] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.155773 13534 raft_consensus.cc:3037] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.158603 13534 raft_consensus.cc:2454] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:21.158618 13404 raft_consensus.cc:2454] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:21.159142 12948 leader_election.cc:302] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, 72b07e9b53b9471989185277045db8f7, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:21.160362 13803 raft_consensus.cc:482] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.160552 13803 raft_consensus.cc:504] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.163014 13803 leader_election.cc:288] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:21.166133 13758 leader_election.cc:288] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 election: Requested vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:21.166703 13758 raft_consensus.cc:2781] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:21.166873 13758 raft_consensus.cc:686] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Becoming Leader. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Running, Role: LEADER
I20240503 14:12:21.167171 13758 consensus_queue.cc:237] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.184855 13144 raft_consensus.cc:2454] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:21.216475 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:21.216753 13404 raft_consensus.cc:2454] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:21.225057 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:21.224726 12740 catalog_manager.cc:5484] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3678714622584fa4883e4bf07fcabd75 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:21.225425 13144 raft_consensus.cc:2454] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:21.226022 13209 leader_election.cc:302] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 7b75d8c9e35a416ba24798fde8b9def0, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:21.227159 13796 raft_consensus.cc:2781] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.227324 13796 raft_consensus.cc:482] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.227432 13796 raft_consensus.cc:3037] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.230959 13796 raft_consensus.cc:504] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.232688 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
I20240503 14:12:21.233049 13143 raft_consensus.cc:3037] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.234027 13806 raft_consensus.cc:482] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.234272 13806 raft_consensus.cc:504] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.236304 13143 raft_consensus.cc:2454] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:21.236913 13806 leader_election.cc:288] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
W20240503 14:12:21.245941 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 86774.4us
I20240503 14:12:21.268432 13796 leader_election.cc:288] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 election: Requested vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:21.268790 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:21.269042 13404 raft_consensus.cc:3037] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.270675 13806 raft_consensus.cc:482] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.270887 13806 raft_consensus.cc:504] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.271941 13404 raft_consensus.cc:2454] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:21.272045 13806 leader_election.cc:288] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:21.272190 13803 raft_consensus.cc:482] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.272367 13803 raft_consensus.cc:504] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.273499 13803 leader_election.cc:288] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
I20240503 14:12:21.275236 13208 leader_election.cc:302] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 7b75d8c9e35a416ba24798fde8b9def0, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:21.277911 13796 raft_consensus.cc:2781] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:21.280284 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:21.281563 13273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:21.281956 13273 raft_consensus.cc:2454] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 0.
I20240503 14:12:21.288578 13817 raft_consensus.cc:482] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.288800 13817 raft_consensus.cc:504] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.290525 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:21.290452 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:21.290869 13143 raft_consensus.cc:2379] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72b07e9b53b9471989185277045db8f7 in current term 1: Already voted for candidate 2cd2acf7d8274b5a95482dffd0823a08 in this term.
I20240503 14:12:21.290894 13144 raft_consensus.cc:2454] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 0.
I20240503 14:12:21.290859 13817 leader_election.cc:288] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:21.291174 13758 raft_consensus.cc:482] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:21.291409 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:21.291393 13758 raft_consensus.cc:504] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.292299 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:21.292526 13143 raft_consensus.cc:2379] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3678714622584fa4883e4bf07fcabd75 in current term 1: Already voted for candidate 2cd2acf7d8274b5a95482dffd0823a08 in this term.
I20240503 14:12:21.292950 13758 leader_election.cc:288] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:12:21.292992 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:21.293200 13143 raft_consensus.cc:2454] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:21.296183 13404 raft_consensus.cc:2379] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b75d8c9e35a416ba24798fde8b9def0 in current term 1: Already voted for candidate 2cd2acf7d8274b5a95482dffd0823a08 in this term.
I20240503 14:12:21.300012 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:21.299957 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:21.300276 13534 raft_consensus.cc:2454] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:21.300376 13014 raft_consensus.cc:2454] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:21.300769 12950 leader_election.cc:302] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0, af88d71da3a54f779cf4f651f4393bb6; no voters:
I20240503 14:12:21.300855 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75"
I20240503 14:12:21.301129 13013 raft_consensus.cc:3037] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.301548 13818 raft_consensus.cc:2781] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:21.301648 13818 raft_consensus.cc:482] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:21.301774 13818 raft_consensus.cc:3037] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.304633 13013 raft_consensus.cc:2454] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:21.304708 13818 raft_consensus.cc:504] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.305707 13818 leader_election.cc:288] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 election: Requested vote from peers af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:12:21.306128 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:21.306334 13534 raft_consensus.cc:3037] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.306391 13273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:21.307000 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:21.306954 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:21.307205 13271 raft_consensus.cc:3037] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.307288 13014 raft_consensus.cc:2379] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b75d8c9e35a416ba24798fde8b9def0 in current term 1: Already voted for candidate 3678714622584fa4883e4bf07fcabd75 in this term.
I20240503 14:12:21.307986 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:21.308205 13013 raft_consensus.cc:2379] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b75d8c9e35a416ba24798fde8b9def0 in current term 1: Already voted for candidate 2cd2acf7d8274b5a95482dffd0823a08 in this term.
I20240503 14:12:21.308537 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:21.308815 13404 raft_consensus.cc:2379] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3678714622584fa4883e4bf07fcabd75 in current term 1: Already voted for candidate 2cd2acf7d8274b5a95482dffd0823a08 in this term.
I20240503 14:12:21.309089 13534 raft_consensus.cc:2454] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:21.309357 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
I20240503 14:12:21.309578 13143 raft_consensus.cc:3037] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.309607 13272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:21.310060 13271 raft_consensus.cc:2454] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:21.312319 13143 raft_consensus.cc:2454] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:21.312943 12949 leader_election.cc:302] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0, af88d71da3a54f779cf4f651f4393bb6; no voters:
I20240503 14:12:21.313323 13818 raft_consensus.cc:2781] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:21.313493 13818 raft_consensus.cc:686] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Becoming Leader. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Running, Role: LEADER
I20240503 14:12:21.313804 13818 consensus_queue.cc:237] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:21.315387 13796 raft_consensus.cc:686] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Running, Role: LEADER
I20240503 14:12:21.316025 12948 leader_election.cc:302] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 7b75d8c9e35a416ba24798fde8b9def0, e148d8a3757340ab9288ea98cabb5ad3
I20240503 14:12:21.315949 13796 consensus_queue.cc:237] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:21.316810 13758 raft_consensus.cc:2726] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:21.317430 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:21.317688 12883 raft_consensus.cc:2454] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 0.
I20240503 14:12:21.317611 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:21.317916 12882 raft_consensus.cc:2454] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 0.
I20240503 14:12:21.318434 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:21.318611 12882 raft_consensus.cc:2454] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:21.319011 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:21.319168 12882 raft_consensus.cc:3037] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.322188 12882 raft_consensus.cc:2454] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:21.322690 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:21.323072 12882 raft_consensus.cc:2379] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3678714622584fa4883e4bf07fcabd75 in current term 1: Already voted for candidate 2cd2acf7d8274b5a95482dffd0823a08 in this term.
I20240503 14:12:21.324290 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
W20240503 14:12:21.324571 13190 tablet.cc:2369] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:21.324625 13534 raft_consensus.cc:2379] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72b07e9b53b9471989185277045db8f7 in current term 1: Already voted for candidate 3678714622584fa4883e4bf07fcabd75 in this term.
I20240503 14:12:21.325947 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:21.326043 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:21.326274 13014 raft_consensus.cc:2379] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72b07e9b53b9471989185277045db8f7 in current term 1: Already voted for candidate 2cd2acf7d8274b5a95482dffd0823a08 in this term.
I20240503 14:12:21.327558 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:21.327855 13534 raft_consensus.cc:2379] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b75d8c9e35a416ba24798fde8b9def0 in current term 1: Already voted for candidate 3678714622584fa4883e4bf07fcabd75 in this term.
I20240503 14:12:21.328490 13078 leader_election.cc:302] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 7b75d8c9e35a416ba24798fde8b9def0; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, e148d8a3757340ab9288ea98cabb5ad3
I20240503 14:12:21.328583 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:21.328956 12882 raft_consensus.cc:2454] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:21.329099 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:21.329319 12883 raft_consensus.cc:3037] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.329928 12740 catalog_manager.cc:5484] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:21.331430 13803 raft_consensus.cc:2726] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:21.332223 12883 raft_consensus.cc:2454] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:21.332276 13077 leader_election.cc:302] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, 7b75d8c9e35a416ba24798fde8b9def0; no voters: 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6
I20240503 14:12:21.332756 13803 raft_consensus.cc:2741] T 1bfebd34071946398632ab7597622a41 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 14:12:21.333987 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:21.334092 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43a1cb5209c4a69a95877c842460adc" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:21.334257 13404 raft_consensus.cc:2379] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72b07e9b53b9471989185277045db8f7 in current term 1: Already voted for candidate 2cd2acf7d8274b5a95482dffd0823a08 in this term.
I20240503 14:12:21.334939 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfebd34071946398632ab7597622a41" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:21.334949 12817 leader_election.cc:302] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 72b07e9b53b9471989185277045db8f7; no voters: 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0, e148d8a3757340ab9288ea98cabb5ad3
I20240503 14:12:21.335201 13274 raft_consensus.cc:2379] T 1bfebd34071946398632ab7597622a41 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72b07e9b53b9471989185277045db8f7 in current term 1: Already voted for candidate 3678714622584fa4883e4bf07fcabd75 in this term.
I20240503 14:12:21.335426 13806 raft_consensus.cc:2726] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:21.335938 12817 leader_election.cc:302] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 72b07e9b53b9471989185277045db8f7, 7b75d8c9e35a416ba24798fde8b9def0; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6
I20240503 14:12:21.336194 13806 raft_consensus.cc:2726] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:21.337426 12740 catalog_manager.cc:5484] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3678714622584fa4883e4bf07fcabd75 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:21.363065 12929 tablet.cc:2369] T 1bfebd34071946398632ab7597622a41 P 72b07e9b53b9471989185277045db8f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:21.597795 13818 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.623967 13818 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.639910 13818 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.647248 13758 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.701246 13796 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.712546 13817 consensus_queue.cc:1035] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:21.722051 13838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:21.722394 13838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:21.724951 13796 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:21.732874 13838 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:21.732918 13835 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.733453 13835 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.743235 13817 consensus_queue.cc:1035] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.759271 13758 consensus_queue.cc:1035] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:21.761355 13758 consensus_queue.cc:1035] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
W20240503 14:12:22.014999 12950 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 56652.3us
INSERT report
rows total: 2000
time total: 357.321 ms
time per row: 0.17866 ms
I20240503 14:12:22.268781 12739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40090:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:22.278206 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 5e130a7b478f4f91af11bef82ebf7236 (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.278735 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e130a7b478f4f91af11bef82ebf7236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.279183 12862 tablet_service.cc:1467] Processing CreateTablet for tablet 5e130a7b478f4f91af11bef82ebf7236 (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.279657 12862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e130a7b478f4f91af11bef82ebf7236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.279698 12863 tablet_service.cc:1467] Processing CreateTablet for tablet 71289cddae864085beacce5aa1fb0c4e (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.280046 12863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71289cddae864085beacce5aa1fb0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.281014 13124 tablet_service.cc:1467] Processing CreateTablet for tablet 5e130a7b478f4f91af11bef82ebf7236 (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.281438 13124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e130a7b478f4f91af11bef82ebf7236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.285753 13123 tablet_service.cc:1467] Processing CreateTablet for tablet 71289cddae864085beacce5aa1fb0c4e (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.286170 13123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71289cddae864085beacce5aa1fb0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.287221 13927 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:22.288416 12994 tablet_service.cc:1467] Processing CreateTablet for tablet 3117228f0f7c431981bf0173b3b3344f (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.288870 12994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3117228f0f7c431981bf0173b3b3344f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.289202 13513 tablet_service.cc:1467] Processing CreateTablet for tablet 5e130a7b478f4f91af11bef82ebf7236 (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.289635 13513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e130a7b478f4f91af11bef82ebf7236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.293937 12863 tablet_service.cc:1467] Processing CreateTablet for tablet 3117228f0f7c431981bf0173b3b3344f (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.294232 13384 tablet_service.cc:1467] Processing CreateTablet for tablet 5e130a7b478f4f91af11bef82ebf7236 (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.294390 12863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3117228f0f7c431981bf0173b3b3344f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.294646 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e130a7b478f4f91af11bef82ebf7236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.297616 13927 tablet_bootstrap.cc:654] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.297677 13383 tablet_service.cc:1467] Processing CreateTablet for tablet 71289cddae864085beacce5aa1fb0c4e (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.298115 13383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71289cddae864085beacce5aa1fb0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.298111 13514 tablet_service.cc:1467] Processing CreateTablet for tablet 71289cddae864085beacce5aa1fb0c4e (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.298509 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71289cddae864085beacce5aa1fb0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.301823 13382 tablet_service.cc:1467] Processing CreateTablet for tablet 3117228f0f7c431981bf0173b3b3344f (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.302078 13253 tablet_service.cc:1467] Processing CreateTablet for tablet 71289cddae864085beacce5aa1fb0c4e (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.302327 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3117228f0f7c431981bf0173b3b3344f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.302480 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71289cddae864085beacce5aa1fb0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.305748 13252 tablet_service.cc:1467] Processing CreateTablet for tablet 3117228f0f7c431981bf0173b3b3344f (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.306252 13252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3117228f0f7c431981bf0173b3b3344f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.310227 13122 tablet_service.cc:1467] Processing CreateTablet for tablet 3117228f0f7c431981bf0173b3b3344f (DEFAULT_TABLE table=rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.310698 13122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3117228f0f7c431981bf0173b3b3344f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.315332 13927 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:22.316049 13927 ts_tablet_manager.cc:1367] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20240503 14:12:22.317180 13927 raft_consensus.cc:348] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.320230 13927 raft_consensus.cc:374] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.320608 13927 raft_consensus.cc:729] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.321233 13927 consensus_queue.cc:260] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.329838 13931 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:22.331598 13927 ts_tablet_manager.cc:1398] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.015s user 0.003s sys 0.001s
I20240503 14:12:22.331923 13931 tablet_bootstrap.cc:654] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.332460 13927 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:22.335657 13929 tablet_bootstrap.cc:492] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:22.337755 13931 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:22.337971 13931 ts_tablet_manager.cc:1367] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:12:22.338860 13927 tablet_bootstrap.cc:654] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.338734 13931 raft_consensus.cc:348] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.339012 13931 raft_consensus.cc:374] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.339087 13931 raft_consensus.cc:729] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.341159 13929 tablet_bootstrap.cc:654] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.342747 13932 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:22.349689 13927 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:22.350061 13927 ts_tablet_manager.cc:1367] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20240503 14:12:22.351028 13927 raft_consensus.cc:348] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.351433 13927 raft_consensus.cc:374] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.351547 13927 raft_consensus.cc:729] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.351909 13927 consensus_queue.cc:260] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.352198 13932 tablet_bootstrap.cc:654] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.353009 13927 ts_tablet_manager.cc:1398] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:22.353420 13927 tablet_bootstrap.cc:492] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:22.358414 13927 tablet_bootstrap.cc:654] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.359083 13928 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:22.360831 13928 tablet_bootstrap.cc:654] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.366238 13926 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:22.366958 13931 consensus_queue.cc:260] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.368496 13926 tablet_bootstrap.cc:654] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.368880 13929 tablet_bootstrap.cc:492] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:22.369050 13929 ts_tablet_manager.cc:1367] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240503 14:12:22.369892 13929 raft_consensus.cc:348] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.370163 13929 raft_consensus.cc:374] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.370244 13929 raft_consensus.cc:729] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.370540 13929 consensus_queue.cc:260] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.374343 13929 ts_tablet_manager.cc:1398] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:22.378698 13932 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:22.378897 13932 ts_tablet_manager.cc:1367] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.006s
I20240503 14:12:22.379753 13932 raft_consensus.cc:348] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.380028 13932 raft_consensus.cc:374] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.380132 13932 raft_consensus.cc:729] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.380422 13932 consensus_queue.cc:260] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.381948 13928 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:22.382124 13928 ts_tablet_manager.cc:1367] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20240503 14:12:22.384642 13928 raft_consensus.cc:348] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.384936 13928 raft_consensus.cc:374] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.385022 13928 raft_consensus.cc:729] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.385318 13928 consensus_queue.cc:260] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.385730 13931 ts_tablet_manager.cc:1398] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.048s user 0.003s sys 0.000s
I20240503 14:12:22.388953 13931 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:22.390738 13931 tablet_bootstrap.cc:654] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.390903 13928 ts_tablet_manager.cc:1398] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.009s user 0.000s sys 0.005s
I20240503 14:12:22.395056 13928 tablet_bootstrap.cc:492] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:22.396384 13932 ts_tablet_manager.cc:1398] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.017s user 0.001s sys 0.004s
I20240503 14:12:22.397768 13932 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:22.398778 13926 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:22.398818 13928 tablet_bootstrap.cc:654] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.398945 13926 ts_tablet_manager.cc:1367] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.018s
I20240503 14:12:22.399477 13943 raft_consensus.cc:482] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.399631 13943 raft_consensus.cc:504] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.399729 13926 raft_consensus.cc:348] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.399957 13926 raft_consensus.cc:374] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.400043 13926 raft_consensus.cc:729] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.400173 13927 tablet_bootstrap.cc:492] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:22.400413 13927 ts_tablet_manager.cc:1367] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.023s
I20240503 14:12:22.400319 13926 consensus_queue.cc:260] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.401093 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:22.401368 13274 raft_consensus.cc:2454] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 0.
I20240503 14:12:22.401625 13932 tablet_bootstrap.cc:654] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.402429 13927 raft_consensus.cc:348] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.402809 13927 raft_consensus.cc:374] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.402794 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:22.402999 13927 raft_consensus.cc:729] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.403017 13404 raft_consensus.cc:2454] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 0.
I20240503 14:12:22.403403 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:22.403501 13078 leader_election.cc:302] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 7b75d8c9e35a416ba24798fde8b9def0, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:22.403615 12883 raft_consensus.cc:2454] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 0.
I20240503 14:12:22.403376 13927 consensus_queue.cc:260] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.404340 13927 ts_tablet_manager.cc:1398] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:22.401115 13926 ts_tablet_manager.cc:1398] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:22.404963 13926 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:22.406742 13926 tablet_bootstrap.cc:654] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.411386 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
W20240503 14:12:22.412315 13080 leader_election.cc:341] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.412813 13943 leader_election.cc:288] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:22.412981 13943 raft_consensus.cc:2781] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:22.413070 13943 raft_consensus.cc:482] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:22.413151 13943 raft_consensus.cc:3037] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.416298 13926 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:22.416591 13926 ts_tablet_manager.cc:1367] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240503 14:12:22.417637 13926 raft_consensus.cc:348] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.418066 13926 raft_consensus.cc:374] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.418183 13926 raft_consensus.cc:729] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.418543 13926 consensus_queue.cc:260] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.419363 13926 ts_tablet_manager.cc:1398] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:22.419721 13926 tablet_bootstrap.cc:492] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:22.422242 13926 tablet_bootstrap.cc:654] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.425442 13943 raft_consensus.cc:504] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.428510 13928 tablet_bootstrap.cc:492] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:22.428705 13928 ts_tablet_manager.cc:1367] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.003s
I20240503 14:12:22.429507 13931 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:22.429693 13931 ts_tablet_manager.cc:1367] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20240503 14:12:22.431206 13926 tablet_bootstrap.cc:492] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:22.431448 13926 ts_tablet_manager.cc:1367] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240503 14:12:22.432245 13943 leader_election.cc:288] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 election: Requested vote from peers 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:22.432312 13926 raft_consensus.cc:348] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.432780 13926 raft_consensus.cc:374] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.432751 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:22.432981 12883 raft_consensus.cc:3037] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.433177 13926 raft_consensus.cc:729] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.433694 13926 consensus_queue.cc:260] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.434237 13796 raft_consensus.cc:482] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.434358 13928 raft_consensus.cc:348] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.434422 13796 raft_consensus.cc:504] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.434638 13928 raft_consensus.cc:374] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.434701 13928 raft_consensus.cc:729] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.435014 13928 consensus_queue.cc:260] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.435484 13796 leader_election.cc:288] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
I20240503 14:12:22.435765 12883 raft_consensus.cc:2454] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 1.
I20240503 14:12:22.436137 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:22.436362 13926 ts_tablet_manager.cc:1398] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:12:22.436376 13014 raft_consensus.cc:2454] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:22.436523 13928 ts_tablet_manager.cc:1398] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:12:22.436753 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:22.436862 13928 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:22.436975 12882 raft_consensus.cc:2454] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:22.437075 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:22.437767 13207 leader_election.cc:302] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, 72b07e9b53b9471989185277045db8f7; no voters:
I20240503 14:12:22.438104 13796 raft_consensus.cc:2781] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:22.438208 13796 raft_consensus.cc:482] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:22.438316 13796 raft_consensus.cc:3037] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.438264 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:22.438491 13404 raft_consensus.cc:3037] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.439307 13928 tablet_bootstrap.cc:654] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.440441 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:22.440665 13274 raft_consensus.cc:3037] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.441089 13404 raft_consensus.cc:2454] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 1.
W20240503 14:12:22.441663 13080 leader_election.cc:341] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.442154 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:22.442265 13928 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:22.442373 13143 raft_consensus.cc:2454] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:22.442651 13928 ts_tablet_manager.cc:1367] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:12:22.443187 13274 raft_consensus.cc:2454] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 1.
I20240503 14:12:22.443329 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:22.443490 13928 raft_consensus.cc:348] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.443708 13928 raft_consensus.cc:374] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.443840 13928 raft_consensus.cc:729] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.443940 13078 leader_election.cc:302] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 7b75d8c9e35a416ba24798fde8b9def0, e148d8a3757340ab9288ea98cabb5ad3; no voters: af88d71da3a54f779cf4f651f4393bb6
W20240503 14:12:22.443974 13208 leader_election.cc:341] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.444139 13928 consensus_queue.cc:260] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.444468 13945 raft_consensus.cc:2781] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.440435 13796 raft_consensus.cc:504] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.445799 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75"
I20240503 14:12:22.446017 13014 raft_consensus.cc:3037] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.432035 13931 raft_consensus.cc:348] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.447816 13931 raft_consensus.cc:374] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.447921 13931 raft_consensus.cc:729] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.448293 13014 raft_consensus.cc:2454] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:22.448204 13931 consensus_queue.cc:260] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.448849 13931 ts_tablet_manager.cc:1398] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20240503 14:12:22.449160 13931 tablet_bootstrap.cc:492] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:22.449447 13928 ts_tablet_manager.cc:1398] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:12:22.450006 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:22.450233 12883 raft_consensus.cc:3037] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.452705 12883 raft_consensus.cc:2454] T 3117228f0f7c431981bf0173b3b3344f P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:22.453161 13207 leader_election.cc:302] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, 72b07e9b53b9471989185277045db8f7; no voters:
I20240503 14:12:22.453240 13796 leader_election.cc:288] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 election: Requested vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
I20240503 14:12:22.453466 13796 raft_consensus.cc:2781] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.453894 13796 raft_consensus.cc:686] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Running, Role: LEADER
I20240503 14:12:22.454222 13796 consensus_queue.cc:237] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.454530 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
I20240503 14:12:22.454736 13143 raft_consensus.cc:3037] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.454972 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
W20240503 14:12:22.455591 13208 leader_election.cc:341] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:22.456688 13932 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:22.456876 13932 ts_tablet_manager.cc:1367] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.015s
I20240503 14:12:22.457263 13931 tablet_bootstrap.cc:654] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.457674 13932 raft_consensus.cc:348] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.457949 13932 raft_consensus.cc:374] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.458020 13932 raft_consensus.cc:729] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.458271 13932 consensus_queue.cc:260] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.458922 13932 ts_tablet_manager.cc:1398] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:22.459285 13143 raft_consensus.cc:2454] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:22.460340 13945 raft_consensus.cc:686] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Becoming Leader. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Running, Role: LEADER
I20240503 14:12:22.460997 13945 consensus_queue.cc:237] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.462625 13931 tablet_bootstrap.cc:492] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:22.462834 13931 ts_tablet_manager.cc:1367] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 14:12:22.463631 13931 raft_consensus.cc:348] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.464202 13931 raft_consensus.cc:374] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.464318 13931 raft_consensus.cc:729] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.464592 13931 consensus_queue.cc:260] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.465785 13931 ts_tablet_manager.cc:1398] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:22.467139 12740 catalog_manager.cc:5484] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3). New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:22.467468 12733 catalog_manager.cc:5484] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:22.584702 13942 raft_consensus.cc:482] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.584908 13942 raft_consensus.cc:504] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.586908 13942 leader_election.cc:288] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:22.608971 13940 raft_consensus.cc:482] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.609225 13940 raft_consensus.cc:504] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.611923 13940 leader_election.cc:288] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:22.640715 13942 raft_consensus.cc:482] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.640951 13942 raft_consensus.cc:504] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.641999 13942 leader_election.cc:288] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:22.649014 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:22.649376 13404 raft_consensus.cc:2454] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:22.649933 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:22.650180 13404 raft_consensus.cc:2379] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af88d71da3a54f779cf4f651f4393bb6 in current term 1: Already voted for candidate 7b75d8c9e35a416ba24798fde8b9def0 in this term.
I20240503 14:12:22.651131 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:22.651388 13274 raft_consensus.cc:2454] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:22.653347 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:22.653621 13274 raft_consensus.cc:2379] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af88d71da3a54f779cf4f651f4393bb6 in current term 1: Already voted for candidate 7b75d8c9e35a416ba24798fde8b9def0 in this term.
I20240503 14:12:22.654099 13468 leader_election.cc:302] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, af88d71da3a54f779cf4f651f4393bb6, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:22.654644 13942 raft_consensus.cc:2781] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:22.654745 13942 raft_consensus.cc:482] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:22.654850 13942 raft_consensus.cc:3037] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.655059 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:22.655294 13143 raft_consensus.cc:2454] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:22.656008 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:22.657001 13468 leader_election.cc:302] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: af88d71da3a54f779cf4f651f4393bb6; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 7b75d8c9e35a416ba24798fde8b9def0, e148d8a3757340ab9288ea98cabb5ad3
I20240503 14:12:22.657801 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:22.657729 13942 raft_consensus.cc:504] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.658062 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e130a7b478f4f91af11bef82ebf7236" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:22.658340 12882 raft_consensus.cc:2379] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af88d71da3a54f779cf4f651f4393bb6 in current term 1: Already voted for candidate 7b75d8c9e35a416ba24798fde8b9def0 in this term.
I20240503 14:12:22.658694 13942 leader_election.cc:288] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:22.658063 12883 raft_consensus.cc:2454] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:22.659065 13942 raft_consensus.cc:3037] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.659415 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:22.659641 12883 raft_consensus.cc:3037] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.662194 13942 raft_consensus.cc:2726] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.662351 12883 raft_consensus.cc:2454] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
I20240503 14:12:22.664758 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:22.665031 13534 raft_consensus.cc:2379] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e148d8a3757340ab9288ea98cabb5ad3 in current term 1: Already voted for candidate af88d71da3a54f779cf4f651f4393bb6 in this term.
I20240503 14:12:22.665653 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
I20240503 14:12:22.665915 13143 raft_consensus.cc:3037] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.666498 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:22.667023 13404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:22.667268 13404 raft_consensus.cc:3037] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.667831 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:22.668097 12883 raft_consensus.cc:2379] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e148d8a3757340ab9288ea98cabb5ad3 in current term 1: Already voted for candidate af88d71da3a54f779cf4f651f4393bb6 in this term.
I20240503 14:12:22.668740 13143 raft_consensus.cc:2454] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
I20240503 14:12:22.668996 13337 leader_election.cc:302] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: e148d8a3757340ab9288ea98cabb5ad3; no voters: 72b07e9b53b9471989185277045db8f7, 7b75d8c9e35a416ba24798fde8b9def0, af88d71da3a54f779cf4f651f4393bb6
I20240503 14:12:22.669381 13469 leader_election.cc:302] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 72b07e9b53b9471989185277045db8f7, 7b75d8c9e35a416ba24798fde8b9def0, af88d71da3a54f779cf4f651f4393bb6; no voters:
I20240503 14:12:22.669651 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:22.669888 13274 raft_consensus.cc:3037] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.669904 13972 raft_consensus.cc:2781] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.670051 13404 raft_consensus.cc:2454] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
I20240503 14:12:22.670089 13972 raft_consensus.cc:686] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Becoming Leader. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Running, Role: LEADER
I20240503 14:12:22.670219 13940 raft_consensus.cc:2726] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:22.671006 13972 consensus_queue.cc:237] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:22.671730 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71289cddae864085beacce5aa1fb0c4e" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:22.672652 13274 raft_consensus.cc:2454] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
I20240503 14:12:22.675048 12739 catalog_manager.cc:5484] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6). New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:22.870865 13945 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:22.877462 13796 consensus_queue.cc:1035] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:22.893988 13940 raft_consensus.cc:482] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:22.894186 13940 raft_consensus.cc:504] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:22.895257 13940 leader_election.cc:288] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
I20240503 14:12:22.896056 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:22.896378 13143 raft_consensus.cc:2379] T 3117228f0f7c431981bf0173b3b3344f P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e148d8a3757340ab9288ea98cabb5ad3 in current term 1: Already voted for candidate 2cd2acf7d8274b5a95482dffd0823a08 in this term.
I20240503 14:12:22.896347 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:22.897337 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:22.898025 13338 leader_election.cc:302] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: e148d8a3757340ab9288ea98cabb5ad3; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, 7b75d8c9e35a416ba24798fde8b9def0
I20240503 14:12:22.898407 13940 raft_consensus.cc:3037] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.901314 13940 raft_consensus.cc:2726] T 3117228f0f7c431981bf0173b3b3344f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:22.902395 12948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.7:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:12:22.904223 12948 consensus_peers.cc:485] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.7:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:12:22.904443 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3117228f0f7c431981bf0173b3b3344f" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:22.904718 13014 raft_consensus.cc:2379] T 3117228f0f7c431981bf0173b3b3344f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e148d8a3757340ab9288ea98cabb5ad3 in current term 1: Already voted for candidate 2cd2acf7d8274b5a95482dffd0823a08 in this term.
I20240503 14:12:22.916082 13945 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:22.920679 13951 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:22.932996 13943 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:22.946839 13796 consensus_queue.cc:1035] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:22.953260 13796 consensus_queue.cc:1035] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:22.961411 13835 consensus_queue.cc:1035] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:23.122495 13972 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:23.129737 13974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:23.130075 13974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:23.138381 13974 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:23.164644 13972 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:23.170019 13972 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:23.170574 13972 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 341.78 ms
time per row: 0.17089 ms
I20240503 14:12:23.642906 12734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40090:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:23.652971 12863 tablet_service.cc:1467] Processing CreateTablet for tablet 32029fdaf54f4bccac238b11019ec913 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.653264 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 32029fdaf54f4bccac238b11019ec913 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.653750 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32029fdaf54f4bccac238b11019ec913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.654094 13514 tablet_service.cc:1467] Processing CreateTablet for tablet 32029fdaf54f4bccac238b11019ec913 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.654527 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32029fdaf54f4bccac238b11019ec913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.656422 13382 tablet_service.cc:1467] Processing CreateTablet for tablet 407844cff18c4421ab36e61e58ab5ee9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.656765 12994 tablet_service.cc:1467] Processing CreateTablet for tablet 32029fdaf54f4bccac238b11019ec913 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.656966 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407844cff18c4421ab36e61e58ab5ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.656985 13384 tablet_service.cc:1467] Processing CreateTablet for tablet a959a6c207af45009ff7807ea6f20ee2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.657172 12994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32029fdaf54f4bccac238b11019ec913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.657482 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a959a6c207af45009ff7807ea6f20ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.657784 13123 tablet_service.cc:1467] Processing CreateTablet for tablet 407844cff18c4421ab36e61e58ab5ee9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.657869 13252 tablet_service.cc:1467] Processing CreateTablet for tablet a959a6c207af45009ff7807ea6f20ee2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.658202 13123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407844cff18c4421ab36e61e58ab5ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.658212 13252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a959a6c207af45009ff7807ea6f20ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.661787 12992 tablet_service.cc:1467] Processing CreateTablet for tablet a959a6c207af45009ff7807ea6f20ee2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.662235 12992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a959a6c207af45009ff7807ea6f20ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.665786 13124 tablet_service.cc:1467] Processing CreateTablet for tablet 32029fdaf54f4bccac238b11019ec913 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.666280 13124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32029fdaf54f4bccac238b11019ec913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.669801 12993 tablet_service.cc:1467] Processing CreateTablet for tablet 407844cff18c4421ab36e61e58ab5ee9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.670286 12993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407844cff18c4421ab36e61e58ab5ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.674044 14065 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:23.674317 13513 tablet_service.cc:1467] Processing CreateTablet for tablet 407844cff18c4421ab36e61e58ab5ee9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.674746 13513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407844cff18c4421ab36e61e58ab5ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.674867 12861 tablet_service.cc:1467] Processing CreateTablet for tablet a959a6c207af45009ff7807ea6f20ee2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.674888 12862 tablet_service.cc:1467] Processing CreateTablet for tablet 407844cff18c4421ab36e61e58ab5ee9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.675339 12862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407844cff18c4421ab36e61e58ab5ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.676657 14065 tablet_bootstrap.cc:654] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.679760 13512 tablet_service.cc:1467] Processing CreateTablet for tablet a959a6c207af45009ff7807ea6f20ee2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:23.680325 13512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a959a6c207af45009ff7807ea6f20ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.680332 14066 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:23.681499 12861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a959a6c207af45009ff7807ea6f20ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.683336 14066 tablet_bootstrap.cc:654] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.685536 12863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32029fdaf54f4bccac238b11019ec913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.690645 14068 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:23.693027 14066 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:23.693198 14066 ts_tablet_manager.cc:1367] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240503 14:12:23.693529 14068 tablet_bootstrap.cc:654] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.694082 14066 raft_consensus.cc:348] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.694342 14066 raft_consensus.cc:374] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.694438 14066 raft_consensus.cc:729] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.694762 14066 consensus_queue.cc:260] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.695664 14066 ts_tablet_manager.cc:1398] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:23.700496 14066 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:23.702353 14066 tablet_bootstrap.cc:654] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.715085 14066 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:23.715296 14066 ts_tablet_manager.cc:1367] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240503 14:12:23.716092 14066 raft_consensus.cc:348] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.716368 14066 raft_consensus.cc:374] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.716450 14066 raft_consensus.cc:729] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.716703 14066 consensus_queue.cc:260] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.718098 14066 ts_tablet_manager.cc:1398] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:23.718416 14066 tablet_bootstrap.cc:492] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:23.720762 14066 tablet_bootstrap.cc:654] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.723809 14072 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:23.725016 14073 tablet_bootstrap.cc:492] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:23.725123 14066 tablet_bootstrap.cc:492] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:23.725301 14066 ts_tablet_manager.cc:1367] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:12:23.726209 14066 raft_consensus.cc:348] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.726505 14066 raft_consensus.cc:374] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.726603 14066 raft_consensus.cc:729] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.726893 14066 consensus_queue.cc:260] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.727471 14073 tablet_bootstrap.cc:654] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.727562 14066 ts_tablet_manager.cc:1398] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:23.729447 14064 tablet_bootstrap.cc:492] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:23.731056 14068 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:23.731267 14068 ts_tablet_manager.cc:1367] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.028s
I20240503 14:12:23.731717 14064 tablet_bootstrap.cc:654] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.732172 14068 raft_consensus.cc:348] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.732527 14068 raft_consensus.cc:374] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.732647 14068 raft_consensus.cc:729] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.733054 14068 consensus_queue.cc:260] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.733593 14065 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:23.733804 14065 ts_tablet_manager.cc:1367] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.060s user 0.019s sys 0.014s
I20240503 14:12:23.733791 14068 ts_tablet_manager.cc:1398] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:23.734500 14068 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:23.734798 14065 raft_consensus.cc:348] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.735123 14065 raft_consensus.cc:374] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.735270 14065 raft_consensus.cc:729] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.735585 14065 consensus_queue.cc:260] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.736378 14065 ts_tablet_manager.cc:1398] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:23.736418 14068 tablet_bootstrap.cc:654] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.736697 14065 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:23.737087 14072 tablet_bootstrap.cc:654] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.747167 14068 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:23.747427 14068 ts_tablet_manager.cc:1367] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20240503 14:12:23.747496 14065 tablet_bootstrap.cc:654] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.747843 14064 tablet_bootstrap.cc:492] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:23.748279 14068 raft_consensus.cc:348] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.748565 14073 tablet_bootstrap.cc:492] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:23.748592 14068 raft_consensus.cc:374] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.748683 14068 raft_consensus.cc:729] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.748816 14073 ts_tablet_manager.cc:1367] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.009s
I20240503 14:12:23.748970 14068 consensus_queue.cc:260] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.749615 14068 ts_tablet_manager.cc:1398] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:23.749976 14068 tablet_bootstrap.cc:492] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:23.749914 14073 raft_consensus.cc:348] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.750231 14073 raft_consensus.cc:374] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.750372 14073 raft_consensus.cc:729] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.750690 14073 consensus_queue.cc:260] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.751101 14064 ts_tablet_manager.cc:1367] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.007s
I20240503 14:12:23.751547 14068 tablet_bootstrap.cc:654] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.752420 14073 ts_tablet_manager.cc:1398] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:23.754177 14072 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:23.755131 14072 ts_tablet_manager.cc:1367] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.007s
I20240503 14:12:23.756462 14072 raft_consensus.cc:348] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.756834 14072 raft_consensus.cc:374] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.756966 14072 raft_consensus.cc:729] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.757328 14072 consensus_queue.cc:260] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.758212 14073 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:23.758460 14064 raft_consensus.cc:348] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.759099 14065 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:23.759300 14065 ts_tablet_manager.cc:1367] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240503 14:12:23.760154 14065 raft_consensus.cc:348] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.760399 14072 ts_tablet_manager.cc:1398] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:23.760538 14065 raft_consensus.cc:374] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.760613 14068 tablet_bootstrap.cc:492] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:23.760673 14065 raft_consensus.cc:729] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.760834 14068 ts_tablet_manager.cc:1367] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240503 14:12:23.761070 14065 consensus_queue.cc:260] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.761219 14064 raft_consensus.cc:374] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.760699 14072 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:23.761654 14068 raft_consensus.cc:348] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.761868 14065 ts_tablet_manager.cc:1398] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:23.761965 14068 raft_consensus.cc:374] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.762094 14068 raft_consensus.cc:729] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.762434 14068 consensus_queue.cc:260] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.763082 14068 ts_tablet_manager.cc:1398] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:23.764389 14072 tablet_bootstrap.cc:654] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.764436 14073 tablet_bootstrap.cc:654] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.765368 14064 raft_consensus.cc:729] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.765812 14064 consensus_queue.cc:260] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.766865 14064 ts_tablet_manager.cc:1398] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:12:23.767182 14064 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:23.767411 14073 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:23.767619 14073 ts_tablet_manager.cc:1367] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 14:12:23.768472 14073 raft_consensus.cc:348] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.768769 14073 raft_consensus.cc:374] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.768878 14073 raft_consensus.cc:729] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.769163 14073 consensus_queue.cc:260] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.769799 14073 ts_tablet_manager.cc:1398] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:23.769560 14072 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:23.770313 14072 ts_tablet_manager.cc:1367] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 14:12:23.770678 14064 tablet_bootstrap.cc:654] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.771039 14072 raft_consensus.cc:348] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.771246 14072 raft_consensus.cc:374] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.771310 14072 raft_consensus.cc:729] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.772118 14072 consensus_queue.cc:260] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.772766 14072 ts_tablet_manager.cc:1398] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:23.774688 14072 tablet_bootstrap.cc:492] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:23.776276 14072 tablet_bootstrap.cc:654] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:23.778440 14072 tablet_bootstrap.cc:492] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:23.778575 14072 ts_tablet_manager.cc:1367] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240503 14:12:23.778569 14064 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:23.778757 14064 ts_tablet_manager.cc:1367] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20240503 14:12:23.779201 14072 raft_consensus.cc:348] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.779392 14072 raft_consensus.cc:374] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.779491 14072 raft_consensus.cc:729] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.779557 14064 raft_consensus.cc:348] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.779719 14072 consensus_queue.cc:260] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.779850 14064 raft_consensus.cc:374] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:23.779971 14064 raft_consensus.cc:729] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.780268 14064 consensus_queue.cc:260] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.780776 14072 ts_tablet_manager.cc:1398] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:23.780918 14064 ts_tablet_manager.cc:1398] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:23.817808 14084 raft_consensus.cc:482] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.818013 14084 raft_consensus.cc:504] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.819238 14084 leader_election.cc:288] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:12:23.819437 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a959a6c207af45009ff7807ea6f20ee2" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:23.819662 13534 raft_consensus.cc:2454] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 0.
I20240503 14:12:23.819614 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a959a6c207af45009ff7807ea6f20ee2" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:23.819852 13014 raft_consensus.cc:2454] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 0.
I20240503 14:12:23.819938 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a959a6c207af45009ff7807ea6f20ee2" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:23.820153 13274 raft_consensus.cc:2454] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 0.
I20240503 14:12:23.820310 13337 leader_election.cc:302] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:23.820658 14084 raft_consensus.cc:2781] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.820783 14084 raft_consensus.cc:482] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.820873 14084 raft_consensus.cc:3037] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.820770 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a959a6c207af45009ff7807ea6f20ee2" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:23.821084 12883 raft_consensus.cc:2454] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 0.
I20240503 14:12:23.823159 14084 raft_consensus.cc:504] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.824039 14084 leader_election.cc:288] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:12:23.824589 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a959a6c207af45009ff7807ea6f20ee2" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75"
I20240503 14:12:23.824604 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a959a6c207af45009ff7807ea6f20ee2" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:23.824793 13534 raft_consensus.cc:3037] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.824808 13014 raft_consensus.cc:3037] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.824764 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a959a6c207af45009ff7807ea6f20ee2" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:23.825009 13274 raft_consensus.cc:3037] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.827564 13014 raft_consensus.cc:2454] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 1.
I20240503 14:12:23.827584 13534 raft_consensus.cc:2454] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 1.
I20240503 14:12:23.828133 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a959a6c207af45009ff7807ea6f20ee2" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:23.828403 12883 raft_consensus.cc:3037] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.828616 13337 leader_election.cc:302] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:23.829001 13274 raft_consensus.cc:2454] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 1.
I20240503 14:12:23.829404 14084 raft_consensus.cc:2781] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.829908 14084 raft_consensus.cc:686] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Becoming Leader. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Running, Role: LEADER
I20240503 14:12:23.830601 14084 consensus_queue.cc:237] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:23.831056 12883 raft_consensus.cc:2454] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 1.
I20240503 14:12:23.834304 14058 raft_consensus.cc:482] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.834484 14058 raft_consensus.cc:504] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.835656 14058 leader_election.cc:288] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:23.836289 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32029fdaf54f4bccac238b11019ec913" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:23.836314 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32029fdaf54f4bccac238b11019ec913" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:23.836537 12883 raft_consensus.cc:2454] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:23.836540 13144 raft_consensus.cc:2454] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:23.836845 12739 catalog_manager.cc:5484] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5). New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:23.837368 13207 leader_election.cc:302] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, 7b75d8c9e35a416ba24798fde8b9def0; no voters:
I20240503 14:12:23.838428 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32029fdaf54f4bccac238b11019ec913" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:23.838654 13014 raft_consensus.cc:2454] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:23.839444 14058 raft_consensus.cc:2781] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.839569 14058 raft_consensus.cc:482] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.839645 14058 raft_consensus.cc:3037] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.842636 14058 raft_consensus.cc:504] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.843994 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32029fdaf54f4bccac238b11019ec913" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
I20240503 14:12:23.844012 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32029fdaf54f4bccac238b11019ec913" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:23.844228 12883 raft_consensus.cc:3037] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.844236 13144 raft_consensus.cc:3037] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.847173 13144 raft_consensus.cc:2454] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:23.847977 12883 raft_consensus.cc:2454] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:23.850016 13209 leader_election.cc:302] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, 7b75d8c9e35a416ba24798fde8b9def0; no voters:
I20240503 14:12:23.850378 13990 raft_consensus.cc:2781] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.850554 13990 raft_consensus.cc:686] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Running, Role: LEADER
I20240503 14:12:23.850914 13990 consensus_queue.cc:237] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:23.852061 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32029fdaf54f4bccac238b11019ec913" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75"
I20240503 14:12:23.852270 13014 raft_consensus.cc:3037] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.852492 14058 leader_election.cc:288] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 1 election: Requested vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:23.854194 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32029fdaf54f4bccac238b11019ec913" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:23.854434 13534 raft_consensus.cc:2454] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 0.
I20240503 14:12:23.854213 13533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32029fdaf54f4bccac238b11019ec913" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:23.854844 13533 raft_consensus.cc:3037] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.854949 13014 raft_consensus.cc:2454] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:23.855026 12739 catalog_manager.cc:5484] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:23.857805 13533 raft_consensus.cc:2454] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd2acf7d8274b5a95482dffd0823a08 in term 1.
I20240503 14:12:23.939585 14081 raft_consensus.cc:482] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:23.939795 14081 raft_consensus.cc:504] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.940871 14081 leader_election.cc:288] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
I20240503 14:12:23.941216 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407844cff18c4421ab36e61e58ab5ee9" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:23.941215 13533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407844cff18c4421ab36e61e58ab5ee9" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:23.941481 13403 raft_consensus.cc:2454] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 0.
I20240503 14:12:23.941483 13533 raft_consensus.cc:2454] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 0.
I20240503 14:12:23.941996 12817 leader_election.cc:302] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 72b07e9b53b9471989185277045db8f7, af88d71da3a54f779cf4f651f4393bb6, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:23.942160 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407844cff18c4421ab36e61e58ab5ee9" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:23.942390 14081 raft_consensus.cc:2781] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:23.942502 14081 raft_consensus.cc:482] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:23.942504 13014 raft_consensus.cc:2454] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 0.
I20240503 14:12:23.942592 14081 raft_consensus.cc:3037] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.942806 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407844cff18c4421ab36e61e58ab5ee9" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:23.943024 13144 raft_consensus.cc:2454] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 0.
I20240503 14:12:23.944772 14081 raft_consensus.cc:504] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.945888 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407844cff18c4421ab36e61e58ab5ee9" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75"
I20240503 14:12:23.946035 13533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407844cff18c4421ab36e61e58ab5ee9" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:23.946018 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407844cff18c4421ab36e61e58ab5ee9" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:23.946179 13014 raft_consensus.cc:3037] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.946229 13403 raft_consensus.cc:3037] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.946269 13533 raft_consensus.cc:3037] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.946557 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407844cff18c4421ab36e61e58ab5ee9" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
I20240503 14:12:23.946774 13144 raft_consensus.cc:3037] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:23.948938 13014 raft_consensus.cc:2454] T 407844cff18c4421ab36e61e58ab5ee9 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 1.
I20240503 14:12:23.948949 13403 raft_consensus.cc:2454] T 407844cff18c4421ab36e61e58ab5ee9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 1.
I20240503 14:12:23.949411 13144 raft_consensus.cc:2454] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 1.
I20240503 14:12:23.949471 12816 leader_election.cc:302] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, 72b07e9b53b9471989185277045db8f7, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:23.949580 13533 raft_consensus.cc:2454] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 1.
I20240503 14:12:23.949890 14081 leader_election.cc:288] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 election: Requested vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
I20240503 14:12:23.950122 14091 raft_consensus.cc:2781] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:23.950307 14091 raft_consensus.cc:686] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [term 1 LEADER]: Becoming Leader. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Running, Role: LEADER
I20240503 14:12:23.950755 14091 consensus_queue.cc:237] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:23.955089 12739 catalog_manager.cc:5484] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72b07e9b53b9471989185277045db8f7 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "72b07e9b53b9471989185277045db8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:24.218724 14084 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.281899 14084 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.287837 14084 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.300176 14084 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.318076 13990 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.343567 13990 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:24.350579 14093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:24.351182 14093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:24.357818 13990 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.361649 13990 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:24.362751 14093 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:24.380883 14091 consensus_queue.cc:1035] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.397209 14091 consensus_queue.cc:1035] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.403959 14081 consensus_queue.cc:1035] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:24.405663 14081 consensus_queue.cc:1035] T 407844cff18c4421ab36e61e58ab5ee9 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 314.892 ms
time per row: 0.157446 ms
I20240503 14:12:24.882881 12739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40090:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:24.895943 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 99d215a7d95a41c6a14e7165cf0fe884 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.895943 13123 tablet_service.cc:1467] Processing CreateTablet for tablet 99d215a7d95a41c6a14e7165cf0fe884 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.896453 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d215a7d95a41c6a14e7165cf0fe884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.896453 13123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d215a7d95a41c6a14e7165cf0fe884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.898062 12994 tablet_service.cc:1467] Processing CreateTablet for tablet 7778260935e54e7d90e6c8d929b2c564 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.898505 12994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7778260935e54e7d90e6c8d929b2c564. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.898727 13382 tablet_service.cc:1467] Processing CreateTablet for tablet 99d215a7d95a41c6a14e7165cf0fe884 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.899123 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d215a7d95a41c6a14e7165cf0fe884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.902082 12992 tablet_service.cc:1467] Processing CreateTablet for tablet 99d215a7d95a41c6a14e7165cf0fe884 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.902542 12992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d215a7d95a41c6a14e7165cf0fe884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.905313 14188 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:24.905364 13513 tablet_service.cc:1467] Processing CreateTablet for tablet f99a260ae30a4efda1126bf96b1ac02f (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.905658 12993 tablet_service.cc:1467] Processing CreateTablet for tablet f99a260ae30a4efda1126bf96b1ac02f (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.905845 13513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99a260ae30a4efda1126bf96b1ac02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.906016 12993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99a260ae30a4efda1126bf96b1ac02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.907096 14188 tablet_bootstrap.cc:654] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.909725 13514 tablet_service.cc:1467] Processing CreateTablet for tablet 99d215a7d95a41c6a14e7165cf0fe884 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.910147 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d215a7d95a41c6a14e7165cf0fe884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.912959 14188 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:24.913110 14188 ts_tablet_manager.cc:1367] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 14:12:24.913942 12861 tablet_service.cc:1467] Processing CreateTablet for tablet 7778260935e54e7d90e6c8d929b2c564 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.913920 14188 raft_consensus.cc:348] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:24.914158 14188 raft_consensus.cc:374] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.914239 14188 raft_consensus.cc:729] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.914398 12861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7778260935e54e7d90e6c8d929b2c564. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.916283 14188 consensus_queue.cc:260] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:24.916910 14188 ts_tablet_manager.cc:1398] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:12:24.918437 13124 tablet_service.cc:1467] Processing CreateTablet for tablet f99a260ae30a4efda1126bf96b1ac02f (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.918877 13124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99a260ae30a4efda1126bf96b1ac02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.921779 13122 tablet_service.cc:1467] Processing CreateTablet for tablet 7778260935e54e7d90e6c8d929b2c564 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.922220 13122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7778260935e54e7d90e6c8d929b2c564. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.927222 13384 tablet_service.cc:1467] Processing CreateTablet for tablet 7778260935e54e7d90e6c8d929b2c564 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.927332 13253 tablet_service.cc:1467] Processing CreateTablet for tablet 7778260935e54e7d90e6c8d929b2c564 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.927294 13252 tablet_service.cc:1467] Processing CreateTablet for tablet f99a260ae30a4efda1126bf96b1ac02f (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.927767 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7778260935e54e7d90e6c8d929b2c564. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.927799 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7778260935e54e7d90e6c8d929b2c564. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.927956 13252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99a260ae30a4efda1126bf96b1ac02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.938014 14188 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:24.939786 14188 tablet_bootstrap.cc:654] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.939692 13382 tablet_service.cc:1467] Processing CreateTablet for tablet f99a260ae30a4efda1126bf96b1ac02f (DEFAULT_TABLE table=rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.940948 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99a260ae30a4efda1126bf96b1ac02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.943549 14191 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:24.948218 14191 tablet_bootstrap.cc:654] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.955605 14191 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:24.956396 14191 ts_tablet_manager.cc:1367] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20240503 14:12:24.959559 14187 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:24.960330 14191 raft_consensus.cc:348] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:24.960716 14191 raft_consensus.cc:374] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.960868 14191 raft_consensus.cc:729] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.961366 14187 tablet_bootstrap.cc:654] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.961207 14191 consensus_queue.cc:260] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:24.962013 14191 ts_tablet_manager.cc:1398] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:24.962291 14191 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:24.964148 14191 tablet_bootstrap.cc:654] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.971166 14193 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:24.974311 14187 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:24.974560 14187 ts_tablet_manager.cc:1367] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240503 14:12:24.974743 14193 tablet_bootstrap.cc:654] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.975505 14187 raft_consensus.cc:348] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:24.975864 14187 raft_consensus.cc:374] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.976006 14187 raft_consensus.cc:729] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.976329 14187 consensus_queue.cc:260] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:24.976997 14187 ts_tablet_manager.cc:1398] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:24.977394 14187 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:24.978381 14188 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:24.978582 14188 ts_tablet_manager.cc:1367] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.009s
I20240503 14:12:24.979434 14190 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:24.979612 14187 tablet_bootstrap.cc:654] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.982125 14190 tablet_bootstrap.cc:654] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.983700 14191 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:24.983541 14188 raft_consensus.cc:348] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:24.983829 14188 raft_consensus.cc:374] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.983911 14188 raft_consensus.cc:729] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.983917 14191 ts_tablet_manager.cc:1367] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20240503 14:12:24.984251 14188 consensus_queue.cc:260] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:24.984915 14188 ts_tablet_manager.cc:1398] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:12:24.985255 14188 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:24.987319 14186 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:24.987295 14191 raft_consensus.cc:348] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:24.987583 14191 raft_consensus.cc:374] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.987675 14191 raft_consensus.cc:729] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.987931 14191 consensus_queue.cc:260] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:24.988503 14191 ts_tablet_manager.cc:1398] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:12:24.989162 14191 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:24.989804 14186 tablet_bootstrap.cc:654] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.990651 14193 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:24.990662 14191 tablet_bootstrap.cc:654] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.990810 14193 ts_tablet_manager.cc:1367] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240503 14:12:24.991588 14098 raft_consensus.cc:482] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:24.991596 14193 raft_consensus.cc:348] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:24.991837 14193 raft_consensus.cc:374] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.991952 14193 raft_consensus.cc:729] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.991762 14098 raft_consensus.cc:504] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:24.992223 14193 consensus_queue.cc:260] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:24.992848 14193 ts_tablet_manager.cc:1398] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:24.992841 14098 leader_election.cc:288] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:12:24.993026 14188 tablet_bootstrap.cc:654] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.994035 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:24.994290 13014 raft_consensus.cc:2454] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 0.
I20240503 14:12:24.994450 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:24.994701 13274 raft_consensus.cc:2454] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 0.
I20240503 14:12:24.994668 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:24.994910 12883 raft_consensus.cc:2454] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 0.
I20240503 14:12:24.995692 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:24.996747 13337 leader_election.cc:302] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, e148d8a3757340ab9288ea98cabb5ad3; no voters:
W20240503 14:12:24.997467 13339 leader_election.cc:341] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:24.998306 14098 raft_consensus.cc:2781] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:24.998442 14098 raft_consensus.cc:482] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:24.998536 14098 raft_consensus.cc:3037] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.998725 14190 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:24.998943 14190 ts_tablet_manager.cc:1367] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240503 14:12:24.999852 14190 raft_consensus.cc:348] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.000216 14190 raft_consensus.cc:374] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.000346 14190 raft_consensus.cc:729] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.000675 14190 consensus_queue.cc:260] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.001585 14190 ts_tablet_manager.cc:1398] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:25.001965 14190 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:25.006142 14190 tablet_bootstrap.cc:654] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.009230 14098 raft_consensus.cc:504] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:25.010694 14190 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:25.010768 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
W20240503 14:12:25.011508 13339 leader_election.cc:341] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:25.011682 14190 ts_tablet_manager.cc:1367] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 14:12:25.014058 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75"
I20240503 14:12:25.014287 13014 raft_consensus.cc:3037] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.014694 14187 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:25.014971 14187 ts_tablet_manager.cc:1367] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.000s
I20240503 14:12:25.016003 14187 raft_consensus.cc:348] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.016405 14187 raft_consensus.cc:374] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.016534 14187 raft_consensus.cc:729] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.016903 14187 consensus_queue.cc:260] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.017683 13014 raft_consensus.cc:2454] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 1.
I20240503 14:12:25.018028 14190 raft_consensus.cc:348] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.018381 14190 raft_consensus.cc:374] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.018503 14190 raft_consensus.cc:729] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.018849 14190 consensus_queue.cc:260] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.019174 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:25.019443 12883 raft_consensus.cc:3037] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.019769 14190 ts_tablet_manager.cc:1398] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:12:25.020833 14098 leader_election.cc:288] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:12:25.017750 14187 ts_tablet_manager.cc:1398] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:25.021308 14187 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:25.022150 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "e148d8a3757340ab9288ea98cabb5ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:25.022377 13274 raft_consensus.cc:3037] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.022933 14191 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:25.022944 14187 tablet_bootstrap.cc:654] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.023118 14191 ts_tablet_manager.cc:1367] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.019s
I20240503 14:12:25.023152 14188 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:25.023383 14188 ts_tablet_manager.cc:1367] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.010s
I20240503 14:12:25.023985 14191 raft_consensus.cc:348] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.024252 14191 raft_consensus.cc:374] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.024370 14191 raft_consensus.cc:729] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.024201 14188 raft_consensus.cc:348] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.024662 14188 raft_consensus.cc:374] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.024623 14191 consensus_queue.cc:260] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.025180 14191 ts_tablet_manager.cc:1398] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:25.024787 14188 raft_consensus.cc:729] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.025939 14188 consensus_queue.cc:260] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.026252 14186 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:25.026494 14186 ts_tablet_manager.cc:1367] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240503 14:12:25.026726 14188 ts_tablet_manager.cc:1398] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:25.027402 14186 raft_consensus.cc:348] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.027936 14186 raft_consensus.cc:374] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.028079 14186 raft_consensus.cc:729] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.028396 14186 consensus_queue.cc:260] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.029546 14186 ts_tablet_manager.cc:1398] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:25.029896 14186 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:25.031855 14186 tablet_bootstrap.cc:654] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.034133 13889 raft_consensus.cc:482] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.034358 13889 raft_consensus.cc:504] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.035542 13889 leader_election.cc:288] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:25.037542 13272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d215a7d95a41c6a14e7165cf0fe884" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:25.037830 13272 raft_consensus.cc:2454] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:25.037861 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d215a7d95a41c6a14e7165cf0fe884" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:25.038139 13534 raft_consensus.cc:2454] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:25.038234 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d215a7d95a41c6a14e7165cf0fe884" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:25.038503 13403 raft_consensus.cc:2454] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:25.039132 12948 leader_election.cc:302] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6; no voters:
I20240503 14:12:25.039496 13889 raft_consensus.cc:2781] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.039638 13889 raft_consensus.cc:482] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.036691 12883 raft_consensus.cc:2454] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 1.
I20240503 14:12:25.039724 13889 raft_consensus.cc:3037] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.040268 13337 leader_election.cc:302] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, 72b07e9b53b9471989185277045db8f7, e148d8a3757340ab9288ea98cabb5ad3; no voters: 7b75d8c9e35a416ba24798fde8b9def0
I20240503 14:12:25.040812 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d215a7d95a41c6a14e7165cf0fe884" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:25.041075 13144 raft_consensus.cc:2454] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:25.042891 13889 raft_consensus.cc:504] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.043069 14098 raft_consensus.cc:2781] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.043231 14098 raft_consensus.cc:686] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Becoming Leader. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Running, Role: LEADER
I20240503 14:12:25.043555 14098 consensus_queue.cc:237] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:25.044812 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d215a7d95a41c6a14e7165cf0fe884" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:25.045061 13403 raft_consensus.cc:3037] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.048449 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d215a7d95a41c6a14e7165cf0fe884" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
I20240503 14:12:25.048722 13144 raft_consensus.cc:3037] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.048697 13403 raft_consensus.cc:2454] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:25.050061 13272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d215a7d95a41c6a14e7165cf0fe884" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:25.050315 13272 raft_consensus.cc:3037] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.052762 13144 raft_consensus.cc:2454] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:25.053246 13272 raft_consensus.cc:2454] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:25.053997 12948 leader_election.cc:302] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:25.055349 14207 raft_consensus.cc:2781] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.055651 13274 raft_consensus.cc:2454] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e148d8a3757340ab9288ea98cabb5ad3 in term 1.
I20240503 14:12:25.055979 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d215a7d95a41c6a14e7165cf0fe884" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:25.056247 13534 raft_consensus.cc:3037] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.059301 13534 raft_consensus.cc:2454] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:25.060077 13889 leader_election.cc:288] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 election: Requested vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:25.061393 14207 raft_consensus.cc:686] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Becoming Leader. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Running, Role: LEADER
I20240503 14:12:25.061771 14207 consensus_queue.cc:237] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } }
I20240503 14:12:25.064491 14186 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:25.064677 14186 ts_tablet_manager.cc:1367] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.035s user 0.019s sys 0.009s
I20240503 14:12:25.064347 12739 catalog_manager.cc:5484] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5). New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:25.065308 14187 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:25.065477 14187 ts_tablet_manager.cc:1367] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.028s
I20240503 14:12:25.065531 14186 raft_consensus.cc:348] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.065773 14186 raft_consensus.cc:374] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.065841 14186 raft_consensus.cc:729] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.066224 14187 raft_consensus.cc:348] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.066499 14187 raft_consensus.cc:374] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.066601 14187 raft_consensus.cc:729] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.066109 14186 consensus_queue.cc:260] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.066819 14187 consensus_queue.cc:260] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.067356 14187 ts_tablet_manager.cc:1398] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:25.068420 14186 ts_tablet_manager.cc:1398] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:25.068732 14186 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:25.070705 14186 tablet_bootstrap.cc:654] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.071856 12739 catalog_manager.cc:5484] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3678714622584fa4883e4bf07fcabd75 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:25.072500 14186 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:25.072631 14186 ts_tablet_manager.cc:1367] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:25.073194 14186 raft_consensus.cc:348] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:25.073477 14186 raft_consensus.cc:374] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.073580 14186 raft_consensus.cc:729] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.073804 14186 consensus_queue.cc:260] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:25.074292 14186 ts_tablet_manager.cc:1398] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:25.139326 13972 raft_consensus.cc:482] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.139555 13972 raft_consensus.cc:504] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.140614 13972 leader_election.cc:288] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:25.140873 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99a260ae30a4efda1126bf96b1ac02f" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:25.141144 13274 raft_consensus.cc:2454] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:25.141618 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99a260ae30a4efda1126bf96b1ac02f" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:25.141857 13144 raft_consensus.cc:2454] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:25.141965 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99a260ae30a4efda1126bf96b1ac02f" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:25.142195 13403 raft_consensus.cc:2454] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:25.142267 13469 leader_election.cc:302] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 7b75d8c9e35a416ba24798fde8b9def0, af88d71da3a54f779cf4f651f4393bb6; no voters:
I20240503 14:12:25.142606 13972 raft_consensus.cc:2781] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:25.142725 13972 raft_consensus.cc:482] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:25.142849 13972 raft_consensus.cc:3037] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.145563 13972 raft_consensus.cc:504] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.146692 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99a260ae30a4efda1126bf96b1ac02f" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:25.146746 13144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99a260ae30a4efda1126bf96b1ac02f" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
I20240503 14:12:25.146934 13274 raft_consensus.cc:3037] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.146953 13144 raft_consensus.cc:3037] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.147927 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99a260ae30a4efda1126bf96b1ac02f" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:25.148160 13403 raft_consensus.cc:3037] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.149452 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99a260ae30a4efda1126bf96b1ac02f" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:25.149695 13014 raft_consensus.cc:2454] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:25.149749 13274 raft_consensus.cc:2454] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
I20240503 14:12:25.150003 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99a260ae30a4efda1126bf96b1ac02f" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75"
I20240503 14:12:25.150166 13013 raft_consensus.cc:3037] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.150880 13403 raft_consensus.cc:2454] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
I20240503 14:12:25.146564 13972 leader_election.cc:288] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 election: Requested vote from peers 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:25.151343 13468 leader_election.cc:302] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, af88d71da3a54f779cf4f651f4393bb6, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:25.151487 13144 raft_consensus.cc:2454] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
I20240503 14:12:25.151762 13972 raft_consensus.cc:2781] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.152281 13972 raft_consensus.cc:686] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Becoming Leader. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Running, Role: LEADER
I20240503 14:12:25.152591 13972 consensus_queue.cc:237] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } }
I20240503 14:12:25.152732 13013 raft_consensus.cc:2454] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
I20240503 14:12:25.156077 12739 catalog_manager.cc:5484] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6). New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:25.180830 13943 raft_consensus.cc:482] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:25.181047 13943 raft_consensus.cc:504] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:25.181967 13943 leader_election.cc:288] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:12:25.182395 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:25.182667 12883 raft_consensus.cc:2379] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b75d8c9e35a416ba24798fde8b9def0 in current term 1: Already voted for candidate e148d8a3757340ab9288ea98cabb5ad3 in this term.
I20240503 14:12:25.182986 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:25.183212 13274 raft_consensus.cc:2379] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b75d8c9e35a416ba24798fde8b9def0 in current term 1: Already voted for candidate e148d8a3757340ab9288ea98cabb5ad3 in this term.
I20240503 14:12:25.183303 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:25.183477 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7778260935e54e7d90e6c8d929b2c564" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:25.183557 13013 raft_consensus.cc:2379] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b75d8c9e35a416ba24798fde8b9def0 in current term 1: Already voted for candidate e148d8a3757340ab9288ea98cabb5ad3 in this term.
I20240503 14:12:25.183938 13077 leader_election.cc:302] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 7b75d8c9e35a416ba24798fde8b9def0; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, 72b07e9b53b9471989185277045db8f7
I20240503 14:12:25.184283 13943 raft_consensus.cc:3037] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.187286 13943 raft_consensus.cc:2726] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:12:25.271397 12948 consensus_peers.cc:485] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.7:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:12:25.447119 14098 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.455610 14098 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.460685 14219 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.463037 14084 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.523555 14208 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:25.546530 14215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:25.546860 14215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:25.555130 14215 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:25.560807 14208 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.565981 14231 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.573388 14207 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240503 14:12:25.609618 13972 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.646876 13972 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.673110 14256 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:25.673503 13972 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 396.767 ms
time per row: 0.198383 ms
-----------------------------------------------------
Suppressions used:
count bytes template
26 6652 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:26.996472 12741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40090:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:27.006553 12861 tablet_service.cc:1467] Processing CreateTablet for tablet a324330d54ff4a01982e13d0cc191296 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.007064 12861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a324330d54ff4a01982e13d0cc191296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.007247 12992 tablet_service.cc:1467] Processing CreateTablet for tablet a324330d54ff4a01982e13d0cc191296 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.007705 12992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a324330d54ff4a01982e13d0cc191296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.008292 13514 tablet_service.cc:1467] Processing CreateTablet for tablet 3d17c42814e24b70b25c24ea15a336f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.008700 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d17c42814e24b70b25c24ea15a336f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.009649 13382 tablet_service.cc:1467] Processing CreateTablet for tablet 3d17c42814e24b70b25c24ea15a336f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.009786 13513 tablet_service.cc:1467] Processing CreateTablet for tablet a324330d54ff4a01982e13d0cc191296 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.010047 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d17c42814e24b70b25c24ea15a336f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.010149 13513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a324330d54ff4a01982e13d0cc191296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.014237 13384 tablet_service.cc:1467] Processing CreateTablet for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.014655 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.015091 13123 tablet_service.cc:1467] Processing CreateTablet for tablet a324330d54ff4a01982e13d0cc191296 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.015501 13123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a324330d54ff4a01982e13d0cc191296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.016502 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.016923 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.017740 13253 tablet_service.cc:1467] Processing CreateTablet for tablet a324330d54ff4a01982e13d0cc191296 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.017882 13122 tablet_service.cc:1467] Processing CreateTablet for tablet 3d17c42814e24b70b25c24ea15a336f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.018159 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a324330d54ff4a01982e13d0cc191296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.018297 13122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d17c42814e24b70b25c24ea15a336f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.021766 13512 tablet_service.cc:1467] Processing CreateTablet for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.022233 13512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.022292 12862 tablet_service.cc:1467] Processing CreateTablet for tablet 3d17c42814e24b70b25c24ea15a336f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.022709 12862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d17c42814e24b70b25c24ea15a336f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.025697 12863 tablet_service.cc:1467] Processing CreateTablet for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.026116 12863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.028909 12993 tablet_service.cc:1467] Processing CreateTablet for tablet 3d17c42814e24b70b25c24ea15a336f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.029402 12993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d17c42814e24b70b25c24ea15a336f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.028967 12994 tablet_service.cc:1467] Processing CreateTablet for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:27.033668 12994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.055300 14303 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:27.057037 14307 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:27.057476 14306 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:27.057631 14308 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:27.059350 14308 tablet_bootstrap.cc:654] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.060804 14307 tablet_bootstrap.cc:654] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.061264 14306 tablet_bootstrap.cc:654] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.061362 14302 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:27.062640 14304 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:27.062989 14302 tablet_bootstrap.cc:654] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.064028 14303 tablet_bootstrap.cc:654] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.064772 14304 tablet_bootstrap.cc:654] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.067921 14307 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:27.067960 14302 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:27.068105 14302 ts_tablet_manager.cc:1367] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:27.068243 14307 ts_tablet_manager.cc:1367] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:12:27.068367 14303 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:27.068521 14303 ts_tablet_manager.cc:1367] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240503 14:12:27.068886 14302 raft_consensus.cc:348] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.069156 14302 raft_consensus.cc:374] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.069258 14302 raft_consensus.cc:729] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.069445 14307 raft_consensus.cc:348] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.069541 14302 consensus_queue.cc:260] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.069620 14303 raft_consensus.cc:348] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.069826 14307 raft_consensus.cc:374] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.069875 14303 raft_consensus.cc:374] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.069985 14303 raft_consensus.cc:729] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.070019 14307 raft_consensus.cc:729] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.070400 14307 consensus_queue.cc:260] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.071281 14307 ts_tablet_manager.cc:1398] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:27.072201 14307 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:27.072355 14302 ts_tablet_manager.cc:1398] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:27.072652 14302 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:27.073268 14304 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:27.073275 14306 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:27.073454 14304 ts_tablet_manager.cc:1367] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 14:12:27.073554 14306 ts_tablet_manager.cc:1367] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20240503 14:12:27.073961 14303 consensus_queue.cc:260] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.074781 14303 ts_tablet_manager.cc:1398] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:27.075409 14306 raft_consensus.cc:348] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.075716 14306 raft_consensus.cc:374] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.075868 14306 raft_consensus.cc:729] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.076164 14306 consensus_queue.cc:260] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.074440 14304 raft_consensus.cc:348] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.076851 14304 raft_consensus.cc:374] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.076959 14304 raft_consensus.cc:729] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.077198 14304 consensus_queue.cc:260] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.077885 14304 ts_tablet_manager.cc:1398] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:27.078176 14304 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08: Bootstrap starting.
I20240503 14:12:27.079752 14304 tablet_bootstrap.cc:654] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.079835 14302 tablet_bootstrap.cc:654] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.082563 14304 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08: No bootstrap required, opened a new log
I20240503 14:12:27.082702 14306 ts_tablet_manager.cc:1398] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:12:27.082741 14304 ts_tablet_manager.cc:1367] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:27.083470 14304 raft_consensus.cc:348] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.083756 14304 raft_consensus.cc:374] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.083833 14304 raft_consensus.cc:729] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd2acf7d8274b5a95482dffd0823a08, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.084107 14304 consensus_queue.cc:260] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.084755 14304 ts_tablet_manager.cc:1398] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:27.085418 14306 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:27.087005 14306 tablet_bootstrap.cc:654] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.088776 14303 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3: Bootstrap starting.
I20240503 14:12:27.089401 14307 tablet_bootstrap.cc:654] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.091444 14308 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:27.091542 14303 tablet_bootstrap.cc:654] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.091617 14308 ts_tablet_manager.cc:1367] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.011s
I20240503 14:12:27.092613 14308 raft_consensus.cc:348] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.092904 14308 raft_consensus.cc:374] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.092985 14308 raft_consensus.cc:729] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.093425 14308 consensus_queue.cc:260] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.094202 14308 ts_tablet_manager.cc:1398] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:27.094525 14308 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0: Bootstrap starting.
I20240503 14:12:27.096573 14308 tablet_bootstrap.cc:654] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.098415 14302 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:27.098615 14302 ts_tablet_manager.cc:1367] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.000s
I20240503 14:12:27.095861 14306 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:27.099256 14306 ts_tablet_manager.cc:1367] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 14:12:27.099586 14302 raft_consensus.cc:348] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.099884 14302 raft_consensus.cc:374] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.099972 14302 raft_consensus.cc:729] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.100286 14306 raft_consensus.cc:348] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.100411 14302 consensus_queue.cc:260] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.100716 14306 raft_consensus.cc:374] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.100839 14306 raft_consensus.cc:729] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.101248 14302 ts_tablet_manager.cc:1398] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:27.101158 14306 consensus_queue.cc:260] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.101707 14302 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
I20240503 14:12:27.103711 14302 tablet_bootstrap.cc:654] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.104907 14306 ts_tablet_manager.cc:1398] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:27.107080 14303 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3: No bootstrap required, opened a new log
I20240503 14:12:27.107280 14303 ts_tablet_manager.cc:1367] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.002s
I20240503 14:12:27.107594 14302 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75: No bootstrap required, opened a new log
I20240503 14:12:27.107748 14302 ts_tablet_manager.cc:1367] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:12:27.108108 14303 raft_consensus.cc:348] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.108371 14303 raft_consensus.cc:374] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.108446 14303 raft_consensus.cc:729] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e148d8a3757340ab9288ea98cabb5ad3, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.108737 14303 consensus_queue.cc:260] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.109503 14303 ts_tablet_manager.cc:1398] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:27.109836 14302 raft_consensus.cc:348] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.110062 14302 raft_consensus.cc:374] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.110138 14302 raft_consensus.cc:729] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.110976 14302 consensus_queue.cc:260] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.112092 14308 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0: No bootstrap required, opened a new log
I20240503 14:12:27.112277 14308 ts_tablet_manager.cc:1367] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240503 14:12:27.113058 14308 raft_consensus.cc:348] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.113358 14308 raft_consensus.cc:374] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.113441 14308 raft_consensus.cc:729] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.113530 14306 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:27.113718 14308 consensus_queue.cc:260] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.116413 14306 tablet_bootstrap.cc:654] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.116824 14308 ts_tablet_manager.cc:1398] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:12:27.118678 14307 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:27.118882 14302 ts_tablet_manager.cc:1398] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:12:27.118944 14307 ts_tablet_manager.cc:1367] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.018s
I20240503 14:12:27.119817 14307 raft_consensus.cc:348] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.120177 14307 raft_consensus.cc:374] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.120360 14307 raft_consensus.cc:729] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.120783 14307 consensus_queue.cc:260] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.121876 14307 ts_tablet_manager.cc:1398] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:27.122231 14307 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6: Bootstrap starting.
I20240503 14:12:27.122468 14306 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7: No bootstrap required, opened a new log
I20240503 14:12:27.122643 14306 ts_tablet_manager.cc:1367] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240503 14:12:27.123487 14306 raft_consensus.cc:348] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.123750 14306 raft_consensus.cc:374] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.123839 14306 raft_consensus.cc:729] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.124091 14306 consensus_queue.cc:260] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.124761 14306 ts_tablet_manager.cc:1398] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:27.124508 14307 tablet_bootstrap.cc:654] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:27.128298 14307 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6: No bootstrap required, opened a new log
I20240503 14:12:27.128557 14307 ts_tablet_manager.cc:1367] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 14:12:27.129390 14307 raft_consensus.cc:348] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.129663 14307 raft_consensus.cc:374] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:27.129770 14307 raft_consensus.cc:729] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.130033 14307 consensus_queue.cc:260] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.131449 14307 ts_tablet_manager.cc:1398] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:27.137535 13943 raft_consensus.cc:482] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.137729 13943 raft_consensus.cc:504] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.138777 13943 leader_election.cc:288] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:27.139690 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d17c42814e24b70b25c24ea15a336f3" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:27.139689 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d17c42814e24b70b25c24ea15a336f3" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:27.139941 13534 raft_consensus.cc:2454] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 0.
I20240503 14:12:27.139942 12883 raft_consensus.cc:2454] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 0.
I20240503 14:12:27.140405 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d17c42814e24b70b25c24ea15a336f3" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:27.140659 13013 raft_consensus.cc:2454] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 0.
I20240503 14:12:27.140688 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d17c42814e24b70b25c24ea15a336f3" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:27.140908 13402 raft_consensus.cc:2454] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 0.
I20240503 14:12:27.141571 13080 leader_election.cc:302] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 4 yes votes; 0 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0, af88d71da3a54f779cf4f651f4393bb6, e148d8a3757340ab9288ea98cabb5ad3; no voters:
I20240503 14:12:27.141909 14324 raft_consensus.cc:2781] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.142027 14324 raft_consensus.cc:482] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.142131 14324 raft_consensus.cc:3037] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.145759 14324 raft_consensus.cc:504] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.146907 14324 leader_election.cc:288] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 election: Requested vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:27.147372 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d17c42814e24b70b25c24ea15a336f3" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:27.147363 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d17c42814e24b70b25c24ea15a336f3" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75"
I20240503 14:12:27.147574 13402 raft_consensus.cc:3037] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.147594 13013 raft_consensus.cc:3037] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.148138 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d17c42814e24b70b25c24ea15a336f3" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:27.148376 12883 raft_consensus.cc:3037] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.150456 13013 raft_consensus.cc:2454] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 1.
I20240503 14:12:27.150462 13402 raft_consensus.cc:2454] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 1.
I20240503 14:12:27.151067 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d17c42814e24b70b25c24ea15a336f3" candidate_uuid: "7b75d8c9e35a416ba24798fde8b9def0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:27.151316 13534 raft_consensus.cc:3037] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.151739 12883 raft_consensus.cc:2454] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 1.
I20240503 14:12:27.152343 13077 leader_election.cc:302] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, 72b07e9b53b9471989185277045db8f7, 7b75d8c9e35a416ba24798fde8b9def0; no voters:
I20240503 14:12:27.152741 14324 raft_consensus.cc:2781] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.152959 14324 raft_consensus.cc:686] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Becoming Leader. State: Replica: 7b75d8c9e35a416ba24798fde8b9def0, State: Running, Role: LEADER
I20240503 14:12:27.153367 14324 consensus_queue.cc:237] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:12:27.155171 13534 raft_consensus.cc:2454] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b75d8c9e35a416ba24798fde8b9def0 in term 1.
I20240503 14:12:27.157418 12737 catalog_manager.cc:5484] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3). New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:27.234962 14162 raft_consensus.cc:482] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.234959 14283 raft_consensus.cc:482] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.235163 14283 raft_consensus.cc:504] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.235160 14162 raft_consensus.cc:504] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.236047 14283 leader_election.cc:288] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
I20240503 14:12:27.236047 14162 leader_election.cc:288] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
I20240503 14:12:27.236864 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:27.236918 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:27.236999 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:27.236999 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:27.237115 12883 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:27.237115 13271 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:27.237216 13013 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 0.
I20240503 14:12:27.237231 13403 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:27.237496 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
I20240503 14:12:27.237661 13468 leader_election.cc:302] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, af88d71da3a54f779cf4f651f4393bb6; no voters:
I20240503 14:12:27.237716 13014 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 0.
I20240503 14:12:27.237746 13272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:27.237923 13272 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 0.
I20240503 14:12:27.237946 14283 raft_consensus.cc:2781] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.238051 14283 raft_consensus.cc:482] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.238137 14283 raft_consensus.cc:3037] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.238291 12817 leader_election.cc:302] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, 72b07e9b53b9471989185277045db8f7; no voters:
I20240503 14:12:27.238384 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:12:27.238579 14162 raft_consensus.cc:2781] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.238586 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:27.238680 14162 raft_consensus.cc:482] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.238760 14162 raft_consensus.cc:3037] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.238726 13402 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 0.
I20240503 14:12:27.240355 14283 raft_consensus.cc:504] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.240729 13534 raft_consensus.cc:2379] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72b07e9b53b9471989185277045db8f7 in current term 1: Already voted for candidate af88d71da3a54f779cf4f651f4393bb6 in this term.
I20240503 14:12:27.240779 14162 raft_consensus.cc:504] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.241353 14283 leader_election.cc:288] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 election: Requested vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
I20240503 14:12:27.241879 13014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75"
I20240503 14:12:27.242100 13014 raft_consensus.cc:3037] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.242147 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75"
I20240503 14:12:27.242125 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:27.242201 13533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:27.242329 13271 raft_consensus.cc:3037] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.242789 14162 leader_election.cc:288] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 election: Requested vote from peers e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
I20240503 14:12:27.243429 13272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:27.244868 13014 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
I20240503 14:12:27.245117 13271 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 1.
I20240503 14:12:27.245630 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:27.245829 12883 raft_consensus.cc:2379] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af88d71da3a54f779cf4f651f4393bb6 in current term 1: Already voted for candidate 72b07e9b53b9471989185277045db8f7 in this term.
I20240503 14:12:27.246371 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "72b07e9b53b9471989185277045db8f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:27.246385 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3"
I20240503 14:12:27.246553 13402 raft_consensus.cc:3037] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.246932 13468 leader_election.cc:302] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, e148d8a3757340ab9288ea98cabb5ad3
I20240503 14:12:27.247509 14283 raft_consensus.cc:2726] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:12:27.248683 13402 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b07e9b53b9471989185277045db8f7 in term 1.
I20240503 14:12:27.249084 12817 leader_election.cc:302] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, e148d8a3757340ab9288ea98cabb5ad3; no voters: 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6
I20240503 14:12:27.249604 14326 raft_consensus.cc:2781] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.249732 14326 raft_consensus.cc:686] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 1 LEADER]: Becoming Leader. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Running, Role: LEADER
I20240503 14:12:27.250047 14326 consensus_queue.cc:237] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:12:27.253383 12737 catalog_manager.cc:5484] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72b07e9b53b9471989185277045db8f7 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "72b07e9b53b9471989185277045db8f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:27.290729 14297 raft_consensus.cc:482] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:27.290937 14297 raft_consensus.cc:504] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.292256 14297 leader_election.cc:288] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:12:27.292397 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a324330d54ff4a01982e13d0cc191296" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0" is_pre_election: true
I20240503 14:12:27.292476 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a324330d54ff4a01982e13d0cc191296" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08" is_pre_election: true
I20240503 14:12:27.292641 13143 raft_consensus.cc:2454] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:27.292690 13271 raft_consensus.cc:2454] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:27.293128 12948 leader_election.cc:302] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0; no voters:
I20240503 14:12:27.293561 14297 raft_consensus.cc:2781] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:27.293671 14297 raft_consensus.cc:482] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:27.293741 14297 raft_consensus.cc:3037] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.293794 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a324330d54ff4a01982e13d0cc191296" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7" is_pre_election: true
I20240503 14:12:27.294045 12883 raft_consensus.cc:2454] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:27.294823 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a324330d54ff4a01982e13d0cc191296" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:12:27.295154 13534 raft_consensus.cc:2454] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 0.
I20240503 14:12:27.296775 14297 raft_consensus.cc:504] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.298311 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a324330d54ff4a01982e13d0cc191296" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd2acf7d8274b5a95482dffd0823a08"
I20240503 14:12:27.298527 13271 raft_consensus.cc:3037] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.298853 13143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a324330d54ff4a01982e13d0cc191296" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b75d8c9e35a416ba24798fde8b9def0"
I20240503 14:12:27.299069 13143 raft_consensus.cc:3037] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.301471 13271 raft_consensus.cc:2454] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:27.301967 14297 leader_election.cc:288] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 election: Requested vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:12:27.302348 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a324330d54ff4a01982e13d0cc191296" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b07e9b53b9471989185277045db8f7"
I20240503 14:12:27.302573 12883 raft_consensus.cc:3037] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.303251 13143 raft_consensus.cc:2454] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:27.303620 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a324330d54ff4a01982e13d0cc191296" candidate_uuid: "3678714622584fa4883e4bf07fcabd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6"
I20240503 14:12:27.303855 13534 raft_consensus.cc:3037] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:27.304279 12949 leader_election.cc:302] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0; no voters:
I20240503 14:12:27.305429 14297 raft_consensus.cc:2781] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:27.305882 14297 raft_consensus.cc:686] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Becoming Leader. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Running, Role: LEADER
I20240503 14:12:27.306233 14297 consensus_queue.cc:237] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } }
I20240503 14:12:27.306622 12883 raft_consensus.cc:2454] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:27.306731 13534 raft_consensus.cc:2454] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3678714622584fa4883e4bf07fcabd75 in term 1.
I20240503 14:12:27.309859 12737 catalog_manager.cc:5484] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3678714622584fa4883e4bf07fcabd75 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:27.487520 12948 consensus_peers.cc:485] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.7:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:12:27.593959 14324 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.635841 14324 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.648162 14335 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:12:27.650876 14335 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.665458 14326 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:27.691213 14329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:27.691587 14329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:27.694298 14326 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.701814 14326 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:27.702507 14329 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:27.704098 14326 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 14:12:27.791103 14297 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.897137 14297 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:27.902215 14297 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:27.905366 12948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.7:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 14:12:27.915382 14297 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:28.018263 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 64238.8us
I20240503 14:12:28.064245 14388 mvcc.cc:205] Tried to move back new op lower bound from 7023597764339511296 to 7023597761770840064. Current Snapshot: MvccSnapshot[applied={T|T < 7023597764339511296}]
W20240503 14:12:28.080063 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 45825us
INSERT report
rows total: 2000
time total: 388.662 ms
time per row: 0.194331 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:29.199424 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.7:44891
--local_ip_for_outbound_sockets=127.10.145.7
--tserver_master_addrs=127.10.145.62:35023
--webserver_port=34589
--webserver_interface=127.10.145.7
--builtin_ntp_servers=127.10.145.20:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:29.563419 14418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:29.563781 14418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:29.563923 14418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:29.572557 14418 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:29.572698 14418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:29.572829 14418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.7
I20240503 14:12:29.582815 14418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.7:44891
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.10.145.7
--webserver_port=34589
--tserver_master_addrs=127.10.145.62:35023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.7
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:29.584300 14418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:29.586215 14418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:29.599124 14429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:29.599521 14426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:29.600523 14427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:29.600782 14418 server_base.cc:1034] running on GCE node
I20240503 14:12:29.601482 14418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:29.602964 14418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:29.604213 14418 hybrid_clock.cc:648] HybridClock initialized: now 1714745549604147 us; error 62 us; skew 500 ppm
I20240503 14:12:29.604560 14418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:29.606940 14418 webserver.cc:460] Webserver started at http://127.10.145.7:34589/ using document root <none> and password file <none>
I20240503 14:12:29.607898 14418 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:29.608033 14418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:29.613538 14418 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 14:12:29.616644 14435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:29.618108 14418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:29.618264 14418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "21d8b64a50ba4c919cb9e711d18fb7de"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-v549"
I20240503 14:12:29.619138 14418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:29.643342 14418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:29.644169 14418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:29.644439 14418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:29.645473 14418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:29.648003 14442 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 14:12:29.654067 14418 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 14:12:29.654218 14418 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20240503 14:12:29.654349 14418 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 14:12:29.658398 14418 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 14:12:29.658561 14418 ts_tablet_manager.cc:569] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:29.658977 14442 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
I20240503 14:12:29.688158 14442 log.cc:824] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:29.691413 14442 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:29.692144 14442 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:12:29.694330 14442 ts_tablet_manager.cc:1367] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.016s
I20240503 14:12:29.706426 14442 raft_consensus.cc:348] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:29.707559 14442 raft_consensus.cc:729] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.708395 14442 consensus_queue.cc:260] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:12:29.710423 14418 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.7:44891
I20240503 14:12:29.710505 14552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.7:44891 every 8 connection(s)
I20240503 14:12:29.710511 14442 ts_tablet_manager.cc:1398] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20240503 14:12:29.712219 14418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
W20240503 14:12:29.713761 14554 tablet.cc:2369] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:29.719512 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 14418
I20240503 14:12:29.719772 10820 test_util.cc:268] Using random seed: 1331628237
I20240503 14:12:29.726949 14553 heartbeater.cc:347] Connected to a master server at 127.10.145.62:35023
I20240503 14:12:29.727363 14553 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:29.728463 14553 heartbeater.cc:511] Master 127.10.145.62:35023 requested a full tablet report, sending...
I20240503 14:12:29.731526 12739 ts_manager.cc:194] Re-registered known tserver with Master: 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
I20240503 14:12:29.733071 12739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.7:33331
I20240503 14:12:29.739737 10820 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 14:12:29.740794 10820 test_util.cc:268] Using random seed: 1331649242
I20240503 14:12:29.758330 10820 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 14:12:29.758706 10820 test_util.cc:268] Using random seed: 1331667160
I20240503 14:12:29.842938 10820 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 14:12:29.913995 10820 test_util.cc:268] Using random seed: 1331822443
I20240503 14:12:29.998772 10820 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 14:12:30.005960 14553 heartbeater.cc:503] Master 127.10.145.62:35023 was elected leader, sending a full tablet report...
I20240503 14:12:30.015141 10820 test_util.cc:268] Using random seed: 1331923567
I20240503 14:12:30.186589 10820 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240503 14:12:30.281447 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 108416us
W20240503 14:12:30.482491 14706 log.cc:924] Time spent T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20240503 14:12:30.688925 13468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 51741.1us
W20240503 14:12:31.189610 13468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 116136us
W20240503 14:12:31.279295 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 46999.6us
W20240503 14:12:31.590932 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 185785us
W20240503 14:12:32.017469 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 101813us
W20240503 14:12:32.417937 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 50430.5us
W20240503 14:12:32.603258 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 46394.5us
W20240503 14:12:33.145818 13337 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 62547.1us
W20240503 14:12:34.227463 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 71006.5us
W20240503 14:12:34.568198 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 50370.1us
W20240503 14:12:35.450582 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 142391us
W20240503 14:12:35.893877 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 121536us
W20240503 14:12:36.585393 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 131336us
W20240503 14:12:37.413111 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 79027.1us
I20240503 14:12:37.667688 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:37.973044 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:38.230221 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.440860 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.474983 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:38.748112 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.782510 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:12:39.885540 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 46894.3us
W20240503 14:12:40.257099 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 244316us
W20240503 14:12:40.832782 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 53145.7us
W20240503 14:12:40.833410 12950 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 52475.1us
W20240503 14:12:40.966046 12949 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 56381.3us
W20240503 14:12:41.058035 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 150895us
I20240503 14:12:43.455212 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:43.491942 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:43.591648 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:44.121208 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.139950 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.154073 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.162050 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:12:44.409663 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 50081.6us
W20240503 14:12:44.577085 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 78578.2us
W20240503 14:12:44.577754 12949 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 77548.5us
W20240503 14:12:44.966104 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 45241us
W20240503 14:12:45.406769 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 100336us
I20240503 14:12:45.878732 13404 consensus_queue.cc:237] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:45.888643 12883 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240503 14:12:45.889214 14785 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 14:12:45.890254 13140 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240503 14:12:45.893807 13271 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240503 14:12:45.895363 14785 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 14:12:45.903635 15061 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:45.917513 15061 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 14:12:45.920656 12877 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:45.922183 13013 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240503 14:12:45.920897 13271 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:45.922736 15061 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 14:12:45.925884 13013 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:45.930290 12737 catalog_manager.cc:5484] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index -1 to 105, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:45.934013 13138 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
W20240503 14:12:45.937528 13338 consensus_peers.cc:485] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7778260935e54e7d90e6c8d929b2c564. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:46.337719 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 51789.9us
W20240503 14:12:46.341770 13339 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 57460.5us
I20240503 14:12:46.656651 15108 ts_tablet_manager.cc:902] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de: Initiating tablet copy from peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:46.658694 13530 consensus_queue.cc:237] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 1.117, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: true } }
W20240503 14:12:46.663072 13467 consensus_peers.cc:485] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 -> Peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Couldn't send request to peer 72b07e9b53b9471989185277045db8f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f99a260ae30a4efda1126bf96b1ac02f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:46.665380 13272 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240503 14:12:46.666184 15014 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240503 14:12:46.668704 13009 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240503 14:12:46.669312 15014 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240503 14:12:46.671928 15014 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.118: config changed from index -1 to 118, NON_VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: true } } }
I20240503 14:12:46.682001 13272 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index -1 to 118, NON_VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: true } } }
I20240503 14:12:46.696969 12742 catalog_manager.cc:5484] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index -1 to 118, NON_VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) added. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: true } } }
I20240503 14:12:46.717758 13142 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240503 14:12:46.718137 13013 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index -1 to 118, NON_VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: true } } }
I20240503 14:12:46.756757 15060 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240503 14:12:46.771724 15108 tablet_copy_client.cc:308] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.5:38005
I20240503 14:12:46.796769 13399 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240503 14:12:46.833405 13143 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index -1 to 118, NON_VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: true } } }
I20240503 14:12:46.840154 13530 consensus_queue.cc:237] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 173, Committed index: 173, Last appended: 1.173, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 174 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:46.850473 13270 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 173. Preceding OpId from leader: term: 1 index: 174. (index mismatch)
I20240503 14:12:46.873736 12883 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 173. Preceding OpId from leader: term: 1 index: 174. (index mismatch)
I20240503 14:12:46.879511 15014 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 174, Last known committed idx: 173, Time since last communication: 0.000s
I20240503 14:12:46.849946 13138 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 173. Preceding OpId from leader: term: 1 index: 174. (index mismatch)
I20240503 14:12:46.893954 15014 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 174, Last known committed idx: 173, Time since last communication: 0.000s
I20240503 14:12:46.906196 14731 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.174: config changed from index -1 to 174, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 174 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:46.909794 15060 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 174, Last known committed idx: 173, Time since last communication: 0.000s
I20240503 14:12:46.921394 13271 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.174: config changed from index -1 to 174, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 174 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:46.924885 12742 catalog_manager.cc:5484] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index -1 to 174, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 174 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:46.933341 12883 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.174: config changed from index -1 to 174, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 174 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:46.946656 13137 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.174: config changed from index -1 to 174, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 174 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:46.974040 14731 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 116, Time since last communication: 0.000s
I20240503 14:12:46.984529 13404 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 174. (index mismatch)
I20240503 14:12:47.011201 15060 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 174, Last known committed idx: 170, Time since last communication: 0.000s
I20240503 14:12:47.140166 13404 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index -1 to 118, NON_VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: true } } }
I20240503 14:12:47.147500 15135 ts_tablet_manager.cc:902] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7: Initiating tablet copy from peer af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:47.148357 13399 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.174: config changed from index -1 to 174, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 174 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:47.154613 13424 tablet_copy_service.cc:140] P e148d8a3757340ab9288ea98cabb5ad3: Received BeginTabletCopySession request for tablet 7778260935e54e7d90e6c8d929b2c564 from peer 21d8b64a50ba4c919cb9e711d18fb7de ({username='slave'} at 127.10.145.7:38721)
I20240503 14:12:47.228067 13424 tablet_copy_service.cc:161] P e148d8a3757340ab9288ea98cabb5ad3: Beginning new tablet copy session on tablet 7778260935e54e7d90e6c8d929b2c564 from peer 21d8b64a50ba4c919cb9e711d18fb7de at {username='slave'} at 127.10.145.7:38721: session id = 21d8b64a50ba4c919cb9e711d18fb7de-7778260935e54e7d90e6c8d929b2c564
W20240503 14:12:47.234891 13468 consensus_peers.cc:485] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71289cddae864085beacce5aa1fb0c4e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:47.242784 13424 tablet_copy_source_session.cc:205] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:47.251179 15108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7778260935e54e7d90e6c8d929b2c564. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:47.355530 13014 consensus_queue.cc:237] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 1.129, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:47.381966 13272 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240503 14:12:47.386097 13529 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240503 14:12:47.386296 15108 tablet_copy_client.cc:783] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:47.386911 15108 tablet_copy_client.cc:647] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:47.388689 14899 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240503 14:12:47.392228 14899 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240503 14:12:47.398442 14899 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.130: config changed from index -1 to 130, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:47.403009 13271 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index -1 to 130, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:47.419076 15108 tablet_copy_client.cc:523] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:47.427206 15153 ts_tablet_manager.cc:902] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: Initiating tablet copy from peer af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:12:47.436165 12742 catalog_manager.cc:5484] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index -1 to 130, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:47.456490 13138 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240503 14:12:47.476774 15108 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
I20240503 14:12:47.475885 14991 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.019s
I20240503 14:12:47.488552 13138 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index -1 to 130, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:47.489895 13400 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240503 14:12:47.482189 15135 tablet_copy_client.cc:308] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36377
W20240503 14:12:47.496337 12948 consensus_peers.cc:485] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99d215a7d95a41c6a14e7165cf0fe884. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:47.497810 14991 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240503 14:12:47.507905 13398 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index -1 to 130, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:47.552771 13532 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index -1 to 130, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:47.586756 13554 tablet_copy_service.cc:140] P af88d71da3a54f779cf4f651f4393bb6: Received BeginTabletCopySession request for tablet f99a260ae30a4efda1126bf96b1ac02f from peer 72b07e9b53b9471989185277045db8f7 ({username='slave'} at 127.10.145.1:41635)
I20240503 14:12:47.587054 13554 tablet_copy_service.cc:161] P af88d71da3a54f779cf4f651f4393bb6: Beginning new tablet copy session on tablet f99a260ae30a4efda1126bf96b1ac02f from peer 72b07e9b53b9471989185277045db8f7 at {username='slave'} at 127.10.145.1:41635: session id = 72b07e9b53b9471989185277045db8f7-f99a260ae30a4efda1126bf96b1ac02f
I20240503 14:12:47.592216 13554 tablet_copy_source_session.cc:205] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:47.596040 15135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99a260ae30a4efda1126bf96b1ac02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:47.627063 15135 tablet_copy_client.cc:783] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:47.627617 15135 tablet_copy_client.cc:647] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7: tablet copy: Starting download of 1 WAL segments...
W20240503 14:12:47.637547 13468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 57351.1us
I20240503 14:12:47.673507 15135 tablet_copy_client.cc:523] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:47.677851 15135 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7: Bootstrap starting.
I20240503 14:12:47.833724 15153 tablet_copy_client.cc:308] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36377
I20240503 14:12:47.992412 13143 consensus_queue.cc:237] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 1.130, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:48.003211 12883 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240503 14:12:48.003875 14950 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240503 14:12:48.015826 13014 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240503 14:12:48.023187 14950 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240503 14:12:48.023806 13532 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240503 14:12:48.025456 14950 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240503 14:12:48.039072 13554 tablet_copy_service.cc:140] P af88d71da3a54f779cf4f651f4393bb6: Received BeginTabletCopySession request for tablet 71289cddae864085beacce5aa1fb0c4e from peer 21d8b64a50ba4c919cb9e711d18fb7de ({username='slave'} at 127.10.145.7:43501)
I20240503 14:12:48.039266 13554 tablet_copy_service.cc:161] P af88d71da3a54f779cf4f651f4393bb6: Beginning new tablet copy session on tablet 71289cddae864085beacce5aa1fb0c4e from peer 21d8b64a50ba4c919cb9e711d18fb7de at {username='slave'} at 127.10.145.7:43501: session id = 21d8b64a50ba4c919cb9e711d18fb7de-71289cddae864085beacce5aa1fb0c4e
I20240503 14:12:48.043562 13554 tablet_copy_source_session.cc:205] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:48.045357 15067 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.131: config changed from index -1 to 131, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:48.061594 12742 catalog_manager.cc:5484] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: config changed from index -1 to 131, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:48.068398 13013 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index -1 to 131, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
W20240503 14:12:48.081970 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 43512.8us
I20240503 14:12:48.102726 12882 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index -1 to 131, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:48.195055 15153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71289cddae864085beacce5aa1fb0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:48.202504 15153 tablet_copy_client.cc:783] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:48.202906 15153 tablet_copy_client.cc:647] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:48.204480 13529 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index -1 to 131, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:48.224066 15186 ts_tablet_manager.cc:902] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de: Initiating tablet copy from peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
I20240503 14:12:48.244423 15153 tablet_copy_client.cc:523] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:48.249971 15153 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
W20240503 14:12:48.284603 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 49224.4us
I20240503 14:12:48.292547 13403 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240503 14:12:48.337777 15067 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240503 14:12:48.344249 13403 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index -1 to 131, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
W20240503 14:12:48.382169 13078 consensus_peers.cc:485] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d17c42814e24b70b25c24ea15a336f3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:48.558377 15199 ts_tablet_manager.cc:902] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de: Initiating tablet copy from peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
W20240503 14:12:48.626061 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 72357.7us
I20240503 14:12:48.578668 15186 tablet_copy_client.cc:308] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:36441
I20240503 14:12:48.604846 15108 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=114 overwritten=0 applied=114 ignored=0} inserts{seen=786 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:48.630803 15108 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:12:48.631172 15108 ts_tablet_manager.cc:1367] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 1.155s user 0.182s sys 0.039s
I20240503 14:12:48.632305 15108 raft_consensus.cc:348] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:48.632653 15108 raft_consensus.cc:729] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: LEARNER
I20240503 14:12:48.632930 15108 consensus_queue.cc:260] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 114, Last appended: 1.114, Last appended by leader: 114, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:48.637135 13403 consensus_queue.cc:237] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 1.216, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:48.658741 15108 ts_tablet_manager.cc:1398] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.027s user 0.002s sys 0.008s
W20240503 14:12:48.662469 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 93703us
I20240503 14:12:48.665977 13424 tablet_copy_service.cc:342] P e148d8a3757340ab9288ea98cabb5ad3: Request end of tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-7778260935e54e7d90e6c8d929b2c564 received from {username='slave'} at 127.10.145.7:38721
I20240503 14:12:48.666183 13424 tablet_copy_service.cc:433] P e148d8a3757340ab9288ea98cabb5ad3: ending tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-7778260935e54e7d90e6c8d929b2c564 on tablet 7778260935e54e7d90e6c8d929b2c564 with peer 21d8b64a50ba4c919cb9e711d18fb7de
I20240503 14:12:48.719626 12882 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20240503 14:12:48.720438 15166 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
W20240503 14:12:48.721639 13338 consensus_peers.cc:485] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a959a6c207af45009ff7807ea6f20ee2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:48.724671 13532 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20240503 14:12:48.725397 15166 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20240503 14:12:48.728267 15166 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.217: config changed from index -1 to 217, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:48.731304 13270 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20240503 14:12:48.731942 14922 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 214, Time since last communication: 0.000s
I20240503 14:12:48.733145 13534 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index -1 to 217, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:48.739583 12737 catalog_manager.cc:5484] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index -1 to 217, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:48.744292 13013 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20240503 14:12:48.745915 15199 tablet_copy_client.cc:308] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:41101
I20240503 14:12:48.756808 12882 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index -1 to 217, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:48.758128 13271 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index -1 to 217, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:48.807178 15166 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
W20240503 14:12:48.888641 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 47606us
W20240503 14:12:48.891829 12947 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 44059us
W20240503 14:12:48.892745 12949 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 46642.6us
I20240503 14:12:48.895095 13011 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index -1 to 217, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:48.906657 13034 tablet_copy_service.cc:140] P 3678714622584fa4883e4bf07fcabd75: Received BeginTabletCopySession request for tablet 99d215a7d95a41c6a14e7165cf0fe884 from peer 21d8b64a50ba4c919cb9e711d18fb7de ({username='slave'} at 127.10.145.7:47587)
I20240503 14:12:48.906991 13034 tablet_copy_service.cc:161] P 3678714622584fa4883e4bf07fcabd75: Beginning new tablet copy session on tablet 99d215a7d95a41c6a14e7165cf0fe884 from peer 21d8b64a50ba4c919cb9e711d18fb7de at {username='slave'} at 127.10.145.7:47587: session id = 21d8b64a50ba4c919cb9e711d18fb7de-99d215a7d95a41c6a14e7165cf0fe884
I20240503 14:12:48.911253 13034 tablet_copy_source_session.cc:205] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 14:12:48.956679 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 49780.7us
I20240503 14:12:48.985574 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d215a7d95a41c6a14e7165cf0fe884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:48.993093 15186 tablet_copy_client.cc:783] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:48.993896 15186 tablet_copy_client.cc:647] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:49.038504 14507 raft_consensus.cc:1203] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.104->[1.105-1.125] Dedup: 1.114->[1.115-1.125]
W20240503 14:12:49.079047 12949 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 73387.9us
W20240503 14:12:49.090373 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 88821us
I20240503 14:12:49.122220 13164 tablet_copy_service.cc:140] P 7b75d8c9e35a416ba24798fde8b9def0: Received BeginTabletCopySession request for tablet 3d17c42814e24b70b25c24ea15a336f3 from peer 21d8b64a50ba4c919cb9e711d18fb7de ({username='slave'} at 127.10.145.7:52389)
I20240503 14:12:49.122514 13164 tablet_copy_service.cc:161] P 7b75d8c9e35a416ba24798fde8b9def0: Beginning new tablet copy session on tablet 3d17c42814e24b70b25c24ea15a336f3 from peer 21d8b64a50ba4c919cb9e711d18fb7de at {username='slave'} at 127.10.145.7:52389: session id = 21d8b64a50ba4c919cb9e711d18fb7de-3d17c42814e24b70b25c24ea15a336f3
I20240503 14:12:49.126670 13164 tablet_copy_source_session.cc:205] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:49.128911 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d17c42814e24b70b25c24ea15a336f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.137068 15199 tablet_copy_client.cc:783] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:49.137468 15199 tablet_copy_client.cc:647] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:49.140362 15199 tablet_copy_client.cc:523] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:49.143954 15199 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
I20240503 14:12:49.197131 15186 tablet_copy_client.cc:523] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:49.201135 15186 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
W20240503 14:12:49.250872 12949 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 48054.3us
I20240503 14:12:49.331084 15247 ts_tablet_manager.cc:902] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: Initiating tablet copy from peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:12:49.343477 15166 raft_consensus.cc:1050] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3: attempting to promote NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de to VOTER
I20240503 14:12:49.344677 15166 consensus_queue.cc:237] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 1.125, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:49.349071 12880 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240503 14:12:49.349676 15134 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240503 14:12:49.350414 13143 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240503 14:12:49.351008 15134 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240503 14:12:49.367977 14506 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240503 14:12:49.370205 13272 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240503 14:12:49.371829 15166 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240503 14:12:49.372794 15166 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240503 14:12:49.386703 15135 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=120 overwritten=0 applied=119 ignored=0} inserts{seen=750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:49.387421 15135 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7: Bootstrap complete.
I20240503 14:12:49.387933 15135 ts_tablet_manager.cc:1367] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7: Time spent bootstrapping tablet: real 1.710s user 0.195s sys 0.018s
I20240503 14:12:49.388849 15135 raft_consensus.cc:348] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: true } }
I20240503 14:12:49.412309 15135 raft_consensus.cc:729] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72b07e9b53b9471989185277045db8f7, State: Initialized, Role: LEARNER
I20240503 14:12:49.412642 15135 consensus_queue.cc:260] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 119, Last appended: 1.120, Last appended by leader: 120, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: true } }
I20240503 14:12:49.414688 15135 ts_tablet_manager.cc:1398] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20240503 14:12:49.416275 13554 tablet_copy_service.cc:342] P af88d71da3a54f779cf4f651f4393bb6: Request end of tablet copy session 72b07e9b53b9471989185277045db8f7-f99a260ae30a4efda1126bf96b1ac02f received from {username='slave'} at 127.10.145.1:41635
I20240503 14:12:49.416456 13554 tablet_copy_service.cc:433] P af88d71da3a54f779cf4f651f4393bb6: ending tablet copy session 72b07e9b53b9471989185277045db8f7-f99a260ae30a4efda1126bf96b1ac02f on tablet f99a260ae30a4efda1126bf96b1ac02f with peer 72b07e9b53b9471989185277045db8f7
I20240503 14:12:49.420782 12883 raft_consensus.cc:1203] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.117->[1.118-1.130] Dedup: 1.120->[1.121-1.130]
I20240503 14:12:49.425449 15134 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.126: config changed from index 105 to 126, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.431205 12879 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 105 to 126, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.434136 12741 catalog_manager.cc:5484] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 reported cstate change: config changed from index 105 to 126, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:49.436537 13138 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 105 to 126, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.443183 13271 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 105 to 126, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.444165 14506 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 105 to 126, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
W20240503 14:12:49.455094 12949 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 72544us
I20240503 14:12:49.466048 13013 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240503 14:12:49.490402 15134 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.004s
I20240503 14:12:49.532931 13011 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 105 to 126, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.555653 15247 tablet_copy_client.cc:308] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.5:38005
I20240503 14:12:49.567670 13424 tablet_copy_service.cc:140] P e148d8a3757340ab9288ea98cabb5ad3: Received BeginTabletCopySession request for tablet a959a6c207af45009ff7807ea6f20ee2 from peer 21d8b64a50ba4c919cb9e711d18fb7de ({username='slave'} at 127.10.145.7:38721)
I20240503 14:12:49.567891 13424 tablet_copy_service.cc:161] P e148d8a3757340ab9288ea98cabb5ad3: Beginning new tablet copy session on tablet a959a6c207af45009ff7807ea6f20ee2 from peer 21d8b64a50ba4c919cb9e711d18fb7de at {username='slave'} at 127.10.145.7:38721: session id = 21d8b64a50ba4c919cb9e711d18fb7de-a959a6c207af45009ff7807ea6f20ee2
I20240503 14:12:49.571084 13424 tablet_copy_source_session.cc:205] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:49.574800 13400 consensus_queue.cc:237] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.126, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:49.580503 15247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a959a6c207af45009ff7807ea6f20ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.590556 15247 tablet_copy_client.cc:783] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:49.591007 15247 tablet_copy_client.cc:647] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:49.602030 15247 tablet_copy_client.cc:523] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:49.603701 12883 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 14:12:49.604139 14506 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 14:12:49.604372 15134 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 14:12:49.605607 13270 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 14:12:49.606810 15134 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 14:12:49.607542 15134 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 14:12:49.607580 15247 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
I20240503 14:12:49.638610 14719 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.127: config changed from index 126 to 127, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.641659 13142 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 14:12:49.645632 12727 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7778260935e54e7d90e6c8d929b2c564 with cas_config_opid_index 126: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:49.646826 14506 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 126 to 127, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.649026 15134 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 14:12:49.653177 12733 catalog_manager.cc:5484] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de reported cstate change: config changed from index 126 to 127, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.661758 13142 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 126 to 127, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.690884 12992 tablet_service.cc:1514] Processing DeleteTablet for tablet 7778260935e54e7d90e6c8d929b2c564 with delete_type TABLET_DATA_TOMBSTONED (TS 3678714622584fa4883e4bf07fcabd75 not found in new config with opid_index 127) from {username='slave'} at 127.0.0.1:55928
I20240503 14:12:49.738513 12880 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 126 to 127, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.749039 13270 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 126 to 127, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:49.820490 15270 tablet_replica.cc:330] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75: stopping tablet replica
I20240503 14:12:49.821100 15270 raft_consensus.cc:2229] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:49.877866 15270 raft_consensus.cc:2258] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:49.878432 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 84621.8us
I20240503 14:12:49.883024 15270 ts_tablet_manager.cc:1875] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:49.935648 15270 ts_tablet_manager.cc:1888] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.126
I20240503 14:12:49.942242 15266 raft_consensus.cc:1050] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6: attempting to promote NON_VOTER 72b07e9b53b9471989185277045db8f7 to VOTER
I20240503 14:12:49.949200 15266 consensus_queue.cc:237] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 1.132, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 133 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } }
I20240503 14:12:49.954190 15148 raft_consensus.cc:1013] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: attempt to promote peer 72b07e9b53b9471989185277045db8f7: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 14:12:49.954686 13137 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 131. Preceding OpId from leader: term: 1 index: 133. (index mismatch)
I20240503 14:12:49.955302 15285 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 133, Last known committed idx: 131, Time since last communication: 0.000s
I20240503 14:12:49.956210 13271 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 132. Preceding OpId from leader: term: 1 index: 133. (index mismatch)
I20240503 14:12:49.957072 15148 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 133, Last known committed idx: 132, Time since last communication: 0.000s
I20240503 14:12:49.958743 13398 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 131. Preceding OpId from leader: term: 1 index: 133. (index mismatch)
I20240503 14:12:49.959393 15148 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 133, Last known committed idx: 131, Time since last communication: 0.000s
I20240503 14:12:49.970402 12881 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 LEARNER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 132. Preceding OpId from leader: term: 1 index: 133. (index mismatch)
I20240503 14:12:49.973201 15150 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 133, Last known committed idx: 131, Time since last communication: 0.000s
I20240503 14:12:49.990749 15148 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.133: config changed from index 118 to 133, 72b07e9b53b9471989185277045db8f7 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 133 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:49.998000 13142 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.133: config changed from index 118 to 133, 72b07e9b53b9471989185277045db8f7 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 133 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:50.005544 12733 catalog_manager.cc:5484] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: config changed from index 118 to 133, 72b07e9b53b9471989185277045db8f7 (127.10.145.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 133 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:50.001811 13272 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.133: config changed from index 118 to 133, 72b07e9b53b9471989185277045db8f7 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 133 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:49.957557 13011 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 132. Preceding OpId from leader: term: 1 index: 133. (index mismatch)
I20240503 14:12:50.029456 15270 log.cc:1196] T 7778260935e54e7d90e6c8d929b2c564 P 3678714622584fa4883e4bf07fcabd75: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/7778260935e54e7d90e6c8d929b2c564
I20240503 14:12:50.031167 13400 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.133: config changed from index 118 to 133, 72b07e9b53b9471989185277045db8f7 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 133 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:50.032608 15284 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 133, Last known committed idx: 131, Time since last communication: 0.000s
I20240503 14:12:50.046452 12726 catalog_manager.cc:4830] TS 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): tablet 7778260935e54e7d90e6c8d929b2c564 (table rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]) successfully deleted
I20240503 14:12:50.052846 13011 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.133: config changed from index 118 to 133, 72b07e9b53b9471989185277045db8f7 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 133 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:50.058955 12880 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.133: config changed from index 118 to 133, 72b07e9b53b9471989185277045db8f7 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 133 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:50.077843 13534 consensus_queue.cc:237] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 134, Committed index: 134, Last appended: 1.134, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } }
I20240503 14:12:50.086342 12883 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240503 14:12:50.087474 13013 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240503 14:12:50.088369 15235 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240503 14:12:50.089748 15235 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
W20240503 14:12:50.091168 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 47986.9us
I20240503 14:12:50.091449 13270 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240503 14:12:50.093394 15235 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 133, Time since last communication: 0.000s
I20240503 14:12:50.128242 15235 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.135: config changed from index 133 to 135, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:50.134429 12729 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f99a260ae30a4efda1126bf96b1ac02f with cas_config_opid_index 133: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:50.138140 13270 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index 133 to 135, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:50.154057 12741 catalog_manager.cc:5484] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index 133 to 135, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:50.159293 13122 tablet_service.cc:1514] Processing DeleteTablet for tablet f99a260ae30a4efda1126bf96b1ac02f with delete_type TABLET_DATA_TOMBSTONED (TS 7b75d8c9e35a416ba24798fde8b9def0 not found in new config with opid_index 135) from {username='slave'} at 127.0.0.1:34920
I20240503 14:12:50.190407 15292 tablet_replica.cc:330] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0: stopping tablet replica
I20240503 14:12:50.198709 15292 raft_consensus.cc:2229] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:50.199642 15292 raft_consensus.cc:2258] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:50.207757 15292 ts_tablet_manager.cc:1875] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:50.233312 12883 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index 133 to 135, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:50.248389 15292 ts_tablet_manager.cc:1888] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.134
I20240503 14:12:50.248719 15292 log.cc:1196] T f99a260ae30a4efda1126bf96b1ac02f P 7b75d8c9e35a416ba24798fde8b9def0: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/f99a260ae30a4efda1126bf96b1ac02f
I20240503 14:12:50.250305 12728 catalog_manager.cc:4830] TS 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): tablet f99a260ae30a4efda1126bf96b1ac02f (table rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]) successfully deleted
I20240503 14:12:50.330195 13013 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index 133 to 135, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
I20240503 14:12:50.366215 13403 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240503 14:12:50.372074 15285 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 133, Time since last communication: 0.000s
I20240503 14:12:50.376813 13403 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index 133 to 135, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } }
W20240503 14:12:50.394789 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 56965.7us
W20240503 14:12:50.409356 13468 consensus_peers.cc:485] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:50.498283 15199 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=135 overwritten=0 applied=135 ignored=0} inserts{seen=766 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:50.548151 15199 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:12:50.548642 15199 ts_tablet_manager.cc:1367] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 1.405s user 0.209s sys 0.044s
I20240503 14:12:50.549635 15199 raft_consensus.cc:348] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:50.550081 15199 raft_consensus.cc:729] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: LEARNER
I20240503 14:12:50.550369 15199 consensus_queue.cc:260] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 135, Last appended: 1.135, Last appended by leader: 135, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:50.619943 15199 ts_tablet_manager.cc:1398] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.071s user 0.000s sys 0.002s
I20240503 14:12:50.622575 13164 tablet_copy_service.cc:342] P 7b75d8c9e35a416ba24798fde8b9def0: Request end of tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-3d17c42814e24b70b25c24ea15a336f3 received from {username='slave'} at 127.10.145.7:52389
I20240503 14:12:50.622789 13164 tablet_copy_service.cc:433] P 7b75d8c9e35a416ba24798fde8b9def0: ending tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-3d17c42814e24b70b25c24ea15a336f3 on tablet 3d17c42814e24b70b25c24ea15a336f3 with peer 21d8b64a50ba4c919cb9e711d18fb7de
W20240503 14:12:50.835109 12947 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 116443us
I20240503 14:12:50.944463 14506 raft_consensus.cc:1203] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.130->[1.131-1.153] Dedup: 1.135->[1.136-1.153]
W20240503 14:12:51.153127 15247 log.cc:924] Time spent T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
W20240503 14:12:51.247987 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 105695us
W20240503 14:12:51.254396 13210 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 92398.4us
W20240503 14:12:51.264078 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 107428us
W20240503 14:12:51.299064 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 44475us
I20240503 14:12:51.369666 14950 raft_consensus.cc:1050] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0: attempting to promote NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de to VOTER
I20240503 14:12:51.371367 14950 consensus_queue.cc:237] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 1.154, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:51.375288 14506 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 155. (index mismatch)
I20240503 14:12:51.376166 15164 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 153, Time since last communication: 0.000s
I20240503 14:12:51.387789 13529 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 155. (index mismatch)
W20240503 14:12:51.393388 15153 log.cc:924] Time spent T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: Append to log took a long time: real 0.114s user 0.003s sys 0.000s
I20240503 14:12:51.396255 12883 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 155. (index mismatch)
I20240503 14:12:51.397534 15216 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 153, Time since last communication: 0.000s
I20240503 14:12:51.398268 15216 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 153, Time since last communication: 0.000s
I20240503 14:12:51.400825 13013 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 155. (index mismatch)
I20240503 14:12:51.431862 15067 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.155: config changed from index 131 to 155, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:51.438192 13529 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.155: config changed from index 131 to 155, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:51.439546 14506 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.155: config changed from index 131 to 155, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:51.440269 12741 catalog_manager.cc:5484] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: config changed from index 131 to 155, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:51.446687 13400 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 155. (index mismatch)
I20240503 14:12:51.450356 15067 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 153, Time since last communication: 0.000s
I20240503 14:12:51.454816 13401 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.155: config changed from index 131 to 155, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:51.463980 12880 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.155: config changed from index 131 to 155, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:51.476305 15067 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 153, Time since last communication: 0.000s
I20240503 14:12:51.519596 13013 raft_consensus.cc:1203] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.153->[1.154-1.155] Dedup: 1.154->[1.155-1.155]
I20240503 14:12:51.520398 13013 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.155: config changed from index 131 to 155, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
W20240503 14:12:51.553782 12948 consensus_peers.cc:485] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:51.793174 13137 consensus_queue.cc:237] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 1.161, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:51.802942 12882 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 161. Preceding OpId from leader: term: 1 index: 162. (index mismatch)
I20240503 14:12:51.808979 15164 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 161, Time since last communication: 0.000s
I20240503 14:12:51.886164 14506 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 161. Preceding OpId from leader: term: 1 index: 162. (index mismatch)
I20240503 14:12:51.934418 13403 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 161. Preceding OpId from leader: term: 1 index: 162. (index mismatch)
I20240503 14:12:51.942510 15067 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 161, Time since last communication: 0.000s
I20240503 14:12:51.943076 15216 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 161, Time since last communication: 0.000s
I20240503 14:12:52.031507 13014 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 161. Preceding OpId from leader: term: 1 index: 162. (index mismatch)
I20240503 14:12:52.032617 15216 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 160, Time since last communication: 0.000s
W20240503 14:12:52.037740 13338 consensus_peers.cc:485] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:52.065971 15067 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.162: config changed from index 155 to 162, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.071595 12728 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d17c42814e24b70b25c24ea15a336f3 with cas_config_opid_index 155: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:52.075562 12733 catalog_manager.cc:5484] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: config changed from index 155 to 162, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:52.075438 12881 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.162: config changed from index 155 to 162, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.102067 13514 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d17c42814e24b70b25c24ea15a336f3 with delete_type TABLET_DATA_TOMBSTONED (TS af88d71da3a54f779cf4f651f4393bb6 not found in new config with opid_index 162) from {username='slave'} at 127.0.0.1:48902
W20240503 14:12:52.121757 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 55920.2us
W20240503 14:12:52.126669 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 76162.8us
I20240503 14:12:52.172168 15186 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=141 overwritten=0 applied=141 ignored=0} inserts{seen=826 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:52.173528 15186 tablet_bootstrap.cc:492] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:12:52.173949 15186 ts_tablet_manager.cc:1367] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 2.973s user 0.221s sys 0.068s
I20240503 14:12:52.175642 15186 raft_consensus.cc:348] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:52.176072 15186 raft_consensus.cc:729] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: LEARNER
I20240503 14:12:52.176345 15186 consensus_queue.cc:260] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 141, Last appended: 1.141, Last appended by leader: 141, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:52.185050 15186 ts_tablet_manager.cc:1398] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240503 14:12:52.223222 14507 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.162: config changed from index 155 to 162, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.274114 15355 tablet_replica.cc:330] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6: stopping tablet replica
I20240503 14:12:52.274614 15355 raft_consensus.cc:2229] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:52.275472 15355 raft_consensus.cc:2258] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:52.277310 15153 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=182 overwritten=0 applied=182 ignored=0} inserts{seen=813 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:52.277983 15153 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:12:52.278316 15153 ts_tablet_manager.cc:1367] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 4.028s user 0.267s sys 0.052s
I20240503 14:12:52.280656 15153 raft_consensus.cc:348] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 174 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:52.281073 15153 raft_consensus.cc:729] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: LEARNER
I20240503 14:12:52.281407 15153 consensus_queue.cc:260] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 182, Last appended: 1.182, Last appended by leader: 182, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 174 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:52.282248 15153 ts_tablet_manager.cc:1398] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:52.285640 13034 tablet_copy_service.cc:342] P 3678714622584fa4883e4bf07fcabd75: Request end of tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-99d215a7d95a41c6a14e7165cf0fe884 received from {username='slave'} at 127.10.145.7:47587
I20240503 14:12:52.285837 13034 tablet_copy_service.cc:433] P 3678714622584fa4883e4bf07fcabd75: ending tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-99d215a7d95a41c6a14e7165cf0fe884 on tablet 99d215a7d95a41c6a14e7165cf0fe884 with peer 21d8b64a50ba4c919cb9e711d18fb7de
I20240503 14:12:52.296084 15355 ts_tablet_manager.cc:1875] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:52.297513 13398 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.162: config changed from index 155 to 162, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.309092 13554 tablet_copy_service.cc:342] P af88d71da3a54f779cf4f651f4393bb6: Request end of tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-71289cddae864085beacce5aa1fb0c4e received from {username='slave'} at 127.10.145.7:43501
I20240503 14:12:52.309260 13554 tablet_copy_service.cc:433] P af88d71da3a54f779cf4f651f4393bb6: ending tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-71289cddae864085beacce5aa1fb0c4e on tablet 71289cddae864085beacce5aa1fb0c4e with peer 21d8b64a50ba4c919cb9e711d18fb7de
W20240503 14:12:52.391424 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 102719us
I20240503 14:12:52.393182 13010 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.162: config changed from index 155 to 162, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.442373 14507 raft_consensus.cc:1203] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.129->[1.130-1.156] Dedup: 1.141->[1.142-1.156]
I20240503 14:12:52.451699 13010 consensus_queue.cc:237] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 182, Committed index: 182, Last appended: 1.182, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:52.460450 13137 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 183. (index mismatch)
I20240503 14:12:52.461333 15331 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240503 14:12:52.466449 12877 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 183. (index mismatch)
I20240503 14:12:52.468297 15331 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240503 14:12:52.471071 15281 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.183: config changed from index -1 to 183, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:52.471736 12877 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index -1 to 183, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:52.478794 12735 catalog_manager.cc:5484] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: config changed from index -1 to 183, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:52.480932 13532 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 183. (index mismatch)
I20240503 14:12:52.484118 15355 ts_tablet_manager.cc:1888] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.161
I20240503 14:12:52.484203 13398 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 183. (index mismatch)
I20240503 14:12:52.484319 15355 log.cc:1196] T 3d17c42814e24b70b25c24ea15a336f3 P af88d71da3a54f779cf4f651f4393bb6: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/3d17c42814e24b70b25c24ea15a336f3
I20240503 14:12:52.485702 12729 catalog_manager.cc:4830] TS af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377): tablet 3d17c42814e24b70b25c24ea15a336f3 (table rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]) successfully deleted
I20240503 14:12:52.486277 13137 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index -1 to 183, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
W20240503 14:12:52.486804 12948 consensus_peers.cc:485] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37542e9a30324860bb575ae02b3c92b9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:52.487907 15091 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240503 14:12:52.491089 13398 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index -1 to 183, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:52.526273 15361 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240503 14:12:52.528868 13529 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index -1 to 183, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:52.570761 15281 raft_consensus.cc:1050] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75: attempting to promote NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de to VOTER
I20240503 14:12:52.572216 15281 consensus_queue.cc:237] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 1.157, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:52.576959 13144 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 14:12:52.578541 14899 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 14:12:52.586481 13529 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 14:12:52.587498 15281 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.001s
I20240503 14:12:52.591085 14506 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 14:12:52.637297 13398 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240503 14:12:52.679582 14504 raft_consensus.cc:1203] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.173->[1.174-1.211] Dedup: 1.182->[1.183-1.211]
W20240503 14:12:52.712388 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 142848us
I20240503 14:12:52.743167 15361 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 14:12:52.743786 15361 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240503 14:12:52.750792 15091 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.158: config changed from index 130 to 158, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.752229 13144 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 130 to 158, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.757154 13403 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 130 to 158, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.757550 12733 catalog_manager.cc:5484] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: config changed from index 130 to 158, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:52.758034 13529 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 130 to 158, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.820899 13273 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
W20240503 14:12:52.826221 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 51960.6us
I20240503 14:12:52.846117 15331 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 14:12:52.851465 14504 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 130 to 158, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.896253 13268 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 130 to 158, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:52.929155 15186 ts_tablet_manager.cc:902] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: Initiating tablet copy from peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
I20240503 14:12:52.967442 13010 consensus_queue.cc:237] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 1.160, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:52.976253 13269 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 14:12:52.976253 13530 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 14:12:52.976914 15091 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 158, Time since last communication: 0.000s
I20240503 14:12:52.977638 14504 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 14:12:52.989006 13144 raft_consensus.cc:1261] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 14:12:52.994282 15331 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I20240503 14:12:53.001713 15361 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I20240503 14:12:53.006888 15331 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.161: config changed from index 158 to 161, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.013149 13531 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 158 to 161, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.018453 13138 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 158 to 161, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.020598 12735 catalog_manager.cc:5484] T 99d215a7d95a41c6a14e7165cf0fe884 P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 158 to 161, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.025907 13382 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d215a7d95a41c6a14e7165cf0fe884 with delete_type TABLET_DATA_TOMBSTONED (TS e148d8a3757340ab9288ea98cabb5ad3 not found in new config with opid_index 161) from {username='slave'} at 127.0.0.1:52634
I20240503 14:12:53.033142 12726 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99d215a7d95a41c6a14e7165cf0fe884 with cas_config_opid_index 158: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:53.030167 15398 tablet_replica.cc:330] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3: stopping tablet replica
I20240503 14:12:53.038429 15398 raft_consensus.cc:2229] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:53.039944 15398 raft_consensus.cc:2258] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:53.043248 15398 ts_tablet_manager.cc:1875] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:53.060587 15398 ts_tablet_manager.cc:1888] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.160
I20240503 14:12:53.060865 15398 log.cc:1196] T 99d215a7d95a41c6a14e7165cf0fe884 P e148d8a3757340ab9288ea98cabb5ad3: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/wals/99d215a7d95a41c6a14e7165cf0fe884
I20240503 14:12:53.061995 12727 catalog_manager.cc:4830] TS e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): tablet 99d215a7d95a41c6a14e7165cf0fe884 (table rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]) successfully deleted
I20240503 14:12:53.068445 15281 consensus_queue.cc:1035] T 99d215a7d95a41c6a14e7165cf0fe884 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I20240503 14:12:53.093659 15285 raft_consensus.cc:1050] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: attempting to promote NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de to VOTER
I20240503 14:12:53.094709 15285 consensus_queue.cc:237] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 1.212, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:53.099217 14507 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240503 14:12:53.099378 12880 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240503 14:12:53.100224 13269 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240503 14:12:53.100600 15235 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20240503 14:12:53.100986 15353 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20240503 14:12:53.101192 15235 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20240503 14:12:53.107954 15353 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.213: config changed from index 174 to 213, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.111793 13144 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240503 14:12:53.112709 15285 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20240503 14:12:53.108918 13269 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 174 to 213, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.116757 13138 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 174 to 213, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.120323 12877 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 174 to 213, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.126987 12733 catalog_manager.cc:5484] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 reported cstate change: config changed from index 174 to 213, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.162029 13273 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 158 to 161, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.174104 14507 raft_consensus.cc:2932] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 158 to 161, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.181133 14504 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 174 to 213, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.187196 13401 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 211. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20240503 14:12:53.209374 15285 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20240503 14:12:53.229871 13403 raft_consensus.cc:1203] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.211->[1.212-1.214] Dedup: 1.212->[1.213-1.214]
I20240503 14:12:53.230988 13403 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 174 to 213, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.265053 15247 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=224 overwritten=0 applied=224 ignored=0} inserts{seen=896 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:53.265565 15247 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:12:53.265866 15247 ts_tablet_manager.cc:1367] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 3.658s user 0.426s sys 0.132s
I20240503 14:12:53.266721 15247 raft_consensus.cc:348] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:53.269410 15247 raft_consensus.cc:729] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: LEARNER
I20240503 14:12:53.269836 15247 consensus_queue.cc:260] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 224, Last appended: 1.224, Last appended by leader: 224, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:53.272015 15247 ts_tablet_manager.cc:1398] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240503 14:12:53.272933 13424 tablet_copy_service.cc:342] P e148d8a3757340ab9288ea98cabb5ad3: Request end of tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-a959a6c207af45009ff7807ea6f20ee2 received from {username='slave'} at 127.10.145.7:38721
I20240503 14:12:53.273078 13424 tablet_copy_service.cc:433] P e148d8a3757340ab9288ea98cabb5ad3: ending tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-a959a6c207af45009ff7807ea6f20ee2 on tablet a959a6c207af45009ff7807ea6f20ee2 with peer 21d8b64a50ba4c919cb9e711d18fb7de
W20240503 14:12:53.314728 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 101534us
W20240503 14:12:53.391320 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 47484.7us
I20240503 14:12:53.412876 14507 raft_consensus.cc:1203] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.216->[1.217-1.250] Dedup: 1.224->[1.225-1.250]
W20240503 14:12:53.428421 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 57370.9us
I20240503 14:12:53.471037 15186 tablet_copy_client.cc:308] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:36441
I20240503 14:12:53.482481 13529 consensus_queue.cc:237] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 1.216, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:53.492003 14504 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20240503 14:12:53.492746 15285 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 215, Time since last communication: 0.000s
I20240503 14:12:53.506606 13034 tablet_copy_service.cc:140] P 3678714622584fa4883e4bf07fcabd75: Received BeginTabletCopySession request for tablet 37542e9a30324860bb575ae02b3c92b9 from peer 21d8b64a50ba4c919cb9e711d18fb7de ({username='slave'} at 127.10.145.7:47587)
I20240503 14:12:53.506831 13034 tablet_copy_service.cc:161] P 3678714622584fa4883e4bf07fcabd75: Beginning new tablet copy session on tablet 37542e9a30324860bb575ae02b3c92b9 from peer 21d8b64a50ba4c919cb9e711d18fb7de at {username='slave'} at 127.10.145.7:47587: session id = 21d8b64a50ba4c919cb9e711d18fb7de-37542e9a30324860bb575ae02b3c92b9
I20240503 14:12:53.512089 13034 tablet_copy_source_session.cc:205] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:53.518710 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37542e9a30324860bb575ae02b3c92b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:53.536756 15186 tablet_copy_client.cc:783] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:53.537439 15186 tablet_copy_client.cc:647] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:53.540851 13144 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20240503 14:12:53.543294 12877 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20240503 14:12:53.543880 13398 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20240503 14:12:53.547447 15186 tablet_copy_client.cc:523] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:53.551903 15353 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20240503 14:12:53.570010 15186 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
I20240503 14:12:53.581483 15419 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 215, Time since last communication: 0.000s
I20240503 14:12:53.590970 15353 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.217: config changed from index 213 to 217, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.595646 12729 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71289cddae864085beacce5aa1fb0c4e with cas_config_opid_index 213: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:53.597232 14504 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index 213 to 217, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.598629 15405 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20240503 14:12:53.598387 12881 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index 213 to 217, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.621563 13398 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index 213 to 217, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.643850 12735 catalog_manager.cc:5484] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 reported cstate change: config changed from index 213 to 217, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.697894 13138 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index 213 to 217, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.918491 15316 raft_consensus.cc:1050] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3: attempting to promote NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de to VOTER
I20240503 14:12:53.919858 15316 consensus_queue.cc:237] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 1.253, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:53.922566 12882 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch)
I20240503 14:12:53.922755 14507 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch)
I20240503 14:12:53.923171 15401 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20240503 14:12:53.923700 15166 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20240503 14:12:53.927585 13010 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch)
I20240503 14:12:53.928489 15166 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20240503 14:12:53.932852 13531 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch)
I20240503 14:12:53.933548 15316 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20240503 14:12:53.936384 15401 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.254: config changed from index 217 to 254, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.937644 13531 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.254: config changed from index 217 to 254, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.938463 13014 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.254: config changed from index 217 to 254, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.941745 12882 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.254: config changed from index 217 to 254, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.962232 13273 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch)
I20240503 14:12:53.962397 14507 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.254: config changed from index 217 to 254, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:53.983852 15401 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.019s
I20240503 14:12:53.993438 13254 tablet_service.cc:1514] Processing DeleteTablet for tablet 71289cddae864085beacce5aa1fb0c4e with delete_type TABLET_DATA_TOMBSTONED (TS 2cd2acf7d8274b5a95482dffd0823a08 not found in new config with opid_index 217) from {username='slave'} at 127.0.0.1:56270
I20240503 14:12:53.994803 13273 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.254: config changed from index 217 to 254, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:54.035244 15451 tablet_replica.cc:330] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: stopping tablet replica
I20240503 14:12:54.038885 15451 raft_consensus.cc:2229] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:54.039280 15451 pending_rounds.cc:62] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: Trying to abort 1 pending ops.
I20240503 14:12:54.039441 15451 pending_rounds.cc:68] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: Aborting op as it isn't in flight: id { term: 1 index: 216 } timestamp: 7023597868870107136 op_type: WRITE_OP write_request { tablet_id: "71289cddae864085beacce5aa1fb0c4e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000|\243:v\2464s\017\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c517374a6e5dbbc4" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023597868591947776 authz_token { token_data: "\010\371\343\323\261\006\"3\n\005slave\022*\n efd4bd07f51242c7a453d29157fb1cc1\020\001\030\001 \001(\001" signature: "<redacted>""\027<6\007\227\004o\022K?\244\254\344\'];\331\336\245q\342B\242v\260\227}\352\035\233\366Hf]\341\246\023l<\027QH\373\276O\264\302\277\310\270\360\265t\262\377\205\206\027\233f\213\377\216&" signing_key_seq_num: 0 } } request_id { client_id: "aaa6899694b747809c724488d5af3f93" seq_no: 627 first_incomplete_seq_no: 627 attempt_no: 0 }
I20240503 14:12:54.040473 15451 raft_consensus.cc:2258] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:54.043354 15451 ts_tablet_manager.cc:1875] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:54.067777 15451 ts_tablet_manager.cc:1888] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.216
I20240503 14:12:54.070921 15451 log.cc:1196] T 71289cddae864085beacce5aa1fb0c4e P 2cd2acf7d8274b5a95482dffd0823a08: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/71289cddae864085beacce5aa1fb0c4e
I20240503 14:12:54.072146 12727 catalog_manager.cc:4830] TS 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): tablet 71289cddae864085beacce5aa1fb0c4e (table rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]) successfully deleted
W20240503 14:12:54.081370 12947 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 55864.6us
I20240503 14:12:54.145807 12739 catalog_manager.cc:5484] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 217 to 254, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:54.160521 13403 consensus_queue.cc:237] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.257, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:54.164002 13529 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240503 14:12:54.165010 15316 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 256, Time since last communication: 0.001s
I20240503 14:12:54.166422 14504 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240503 14:12:54.166999 13268 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240503 14:12:54.167069 15316 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 256, Time since last communication: 0.000s
I20240503 14:12:54.167726 15401 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 256, Time since last communication: 0.000s
I20240503 14:12:54.178123 15316 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.258: config changed from index 254 to 258, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:54.184386 13273 raft_consensus.cc:1203] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.256->[1.257-1.257] Dedup: 1.257->[]
I20240503 14:12:54.184201 14504 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 254 to 258, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:54.186331 13529 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 254 to 258, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:54.188309 12727 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a959a6c207af45009ff7807ea6f20ee2 with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:54.188614 13268 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 254 to 258, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:54.194583 12733 catalog_manager.cc:5484] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 254 to 258, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:54.202670 12879 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240503 14:12:54.215677 12992 tablet_service.cc:1514] Processing DeleteTablet for tablet a959a6c207af45009ff7807ea6f20ee2 with delete_type TABLET_DATA_TOMBSTONED (TS 3678714622584fa4883e4bf07fcabd75 not found in new config with opid_index 258) from {username='slave'} at 127.0.0.1:55928
I20240503 14:12:54.221611 15460 tablet_replica.cc:330] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75: stopping tablet replica
I20240503 14:12:54.221834 15460 raft_consensus.cc:2229] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:54.222048 15460 raft_consensus.cc:2258] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:54.226037 15316 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 256, Time since last communication: 0.002s
I20240503 14:12:54.228951 12879 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 254 to 258, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:54.281965 15460 ts_tablet_manager.cc:1875] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:54.452188 15460 ts_tablet_manager.cc:1888] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.255
I20240503 14:12:54.452353 15460 log.cc:1196] T a959a6c207af45009ff7807ea6f20ee2 P 3678714622584fa4883e4bf07fcabd75: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/a959a6c207af45009ff7807ea6f20ee2
I20240503 14:12:54.469817 12726 catalog_manager.cc:4830] TS 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): tablet a959a6c207af45009ff7807ea6f20ee2 (table rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]) successfully deleted
W20240503 14:12:54.541954 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 74265us
I20240503 14:12:54.610736 13009 consensus_queue.cc:237] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 1.160, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:54.617841 13534 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 14:12:54.618530 15455 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I20240503 14:12:54.621217 13143 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 14:12:54.622156 15455 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I20240503 14:12:54.637399 15331 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.161: config changed from index -1 to 161, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:54.638182 13529 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index -1 to 161, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:54.643034 13143 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index -1 to 161, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:54.643747 12734 catalog_manager.cc:5484] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: config changed from index -1 to 161, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:54.684441 13269 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
W20240503 14:12:54.707674 13339 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 52137.9us
I20240503 14:12:54.753732 12877 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
W20240503 14:12:54.823890 12948 consensus_peers.cc:485] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a324330d54ff4a01982e13d0cc191296. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:54.827515 14899 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I20240503 14:12:54.831416 15331 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I20240503 14:12:54.901518 13268 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index -1 to 161, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:54.927376 12879 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index -1 to 161, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
W20240503 14:12:55.161821 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 130440us
I20240503 14:12:55.193372 15485 ts_tablet_manager.cc:902] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de: Initiating tablet copy from peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
I20240503 14:12:55.264981 15485 tablet_copy_client.cc:308] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:36441
I20240503 14:12:55.395401 13034 tablet_copy_service.cc:140] P 3678714622584fa4883e4bf07fcabd75: Received BeginTabletCopySession request for tablet a324330d54ff4a01982e13d0cc191296 from peer 21d8b64a50ba4c919cb9e711d18fb7de ({username='slave'} at 127.10.145.7:47587)
I20240503 14:12:55.395642 13034 tablet_copy_service.cc:161] P 3678714622584fa4883e4bf07fcabd75: Beginning new tablet copy session on tablet a324330d54ff4a01982e13d0cc191296 from peer 21d8b64a50ba4c919cb9e711d18fb7de at {username='slave'} at 127.10.145.7:47587: session id = 21d8b64a50ba4c919cb9e711d18fb7de-a324330d54ff4a01982e13d0cc191296
I20240503 14:12:55.400041 13034 tablet_copy_source_session.cc:205] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:55.434883 15485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a324330d54ff4a01982e13d0cc191296. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.455691 15485 tablet_copy_client.cc:783] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:55.455991 15485 tablet_copy_client.cc:647] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:55.461712 15485 tablet_copy_client.cc:523] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:55.466364 15485 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
W20240503 14:12:55.672570 12818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 170072us
I20240503 14:12:55.936074 15485 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=162 overwritten=0 applied=162 ignored=0} inserts{seen=847 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:55.936923 15485 tablet_bootstrap.cc:492] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:12:55.937949 15485 ts_tablet_manager.cc:1367] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 0.472s user 0.232s sys 0.034s
I20240503 14:12:55.939591 15485 raft_consensus.cc:348] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:55.940145 15485 raft_consensus.cc:729] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: LEARNER
I20240503 14:12:55.940498 15485 consensus_queue.cc:260] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 162, Last appended: 1.162, Last appended by leader: 162, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:55.949319 15485 ts_tablet_manager.cc:1398] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.011s user 0.002s sys 0.006s
I20240503 14:12:55.961972 13034 tablet_copy_service.cc:342] P 3678714622584fa4883e4bf07fcabd75: Request end of tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-a324330d54ff4a01982e13d0cc191296 received from {username='slave'} at 127.10.145.7:47587
I20240503 14:12:55.962214 13034 tablet_copy_service.cc:433] P 3678714622584fa4883e4bf07fcabd75: ending tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-a324330d54ff4a01982e13d0cc191296 on tablet a324330d54ff4a01982e13d0cc191296 with peer 21d8b64a50ba4c919cb9e711d18fb7de
I20240503 14:12:56.039252 13137 consensus_queue.cc:237] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 1.252, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:56.069005 13529 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240503 14:12:56.072863 15492 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240503 14:12:56.078519 13401 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240503 14:12:56.107321 12879 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240503 14:12:56.108383 15501 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 251, Time since last communication: 0.000s
I20240503 14:12:56.120944 15492 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.253: config changed from index -1 to 253, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:56.124114 13273 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
W20240503 14:12:56.126396 13078 consensus_peers.cc:485] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e130a7b478f4f91af11bef82ebf7236. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:56.133879 15492 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240503 14:12:56.134320 13529 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index -1 to 253, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:56.152221 12733 catalog_manager.cc:5484] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: config changed from index -1 to 253, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:56.169201 15501 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 251, Time since last communication: 0.000s
I20240503 14:12:56.179615 13403 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index -1 to 253, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:56.182006 13273 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index -1 to 253, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:12:56.214708 14507 raft_consensus.cc:1203] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.160->[1.161-1.169] Dedup: 1.162->[1.163-1.169]
I20240503 14:12:56.307132 12879 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index -1 to 253, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
W20240503 14:12:56.386804 12948 consensus_peers.cc:485] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 14:12:56.421682 12816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 81966.9us
I20240503 14:12:56.430984 15331 raft_consensus.cc:1050] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75: attempting to promote NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de to VOTER
I20240503 14:12:56.431979 15331 consensus_queue.cc:237] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 171, Committed index: 171, Last appended: 1.171, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:56.443246 13534 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 171. Preceding OpId from leader: term: 1 index: 172. (index mismatch)
I20240503 14:12:56.444110 15281 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
W20240503 14:12:56.471575 15186 log.cc:924] Time spent T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: Append to log took a long time: real 0.172s user 0.001s sys 0.000s
I20240503 14:12:56.500761 12876 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 171. Preceding OpId from leader: term: 1 index: 172. (index mismatch)
I20240503 14:12:56.503969 15331 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 170, Time since last communication: 0.000s
I20240503 14:12:56.517835 13144 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 171. Preceding OpId from leader: term: 1 index: 172. (index mismatch)
I20240503 14:12:56.524129 15455 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
I20240503 14:12:56.565094 15525 ts_tablet_manager.cc:902] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: Initiating tablet copy from peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
I20240503 14:12:56.571381 13268 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 171. Preceding OpId from leader: term: 1 index: 172. (index mismatch)
I20240503 14:12:56.590502 15503 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
I20240503 14:12:56.625417 15503 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.172: config changed from index 161 to 172, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
W20240503 14:12:56.633067 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 77051.3us
I20240503 14:12:56.643884 13269 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.172: config changed from index 161 to 172, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:56.711037 12737 catalog_manager.cc:5484] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: config changed from index 161 to 172, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:56.712940 13530 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.172: config changed from index 161 to 172, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:56.740237 13140 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.172: config changed from index 161 to 172, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:56.751673 15525 tablet_copy_client.cc:308] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:41101
I20240503 14:12:56.753371 13164 tablet_copy_service.cc:140] P 7b75d8c9e35a416ba24798fde8b9def0: Received BeginTabletCopySession request for tablet 5e130a7b478f4f91af11bef82ebf7236 from peer 21d8b64a50ba4c919cb9e711d18fb7de ({username='slave'} at 127.10.145.7:52389)
I20240503 14:12:56.753607 13164 tablet_copy_service.cc:161] P 7b75d8c9e35a416ba24798fde8b9def0: Beginning new tablet copy session on tablet 5e130a7b478f4f91af11bef82ebf7236 from peer 21d8b64a50ba4c919cb9e711d18fb7de at {username='slave'} at 127.10.145.7:52389: session id = 21d8b64a50ba4c919cb9e711d18fb7de-5e130a7b478f4f91af11bef82ebf7236
I20240503 14:12:56.756469 13164 tablet_copy_source_session.cc:205] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:56.759697 15525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e130a7b478f4f91af11bef82ebf7236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:56.787165 15525 tablet_copy_client.cc:783] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:56.787950 15525 tablet_copy_client.cc:647] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:56.792826 15525 tablet_copy_client.cc:523] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:56.796792 15525 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
W20240503 14:12:56.816483 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 128773us
I20240503 14:12:56.913249 12879 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.172: config changed from index 161 to 172, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
W20240503 14:12:56.956329 15186 log.cc:924] Time spent T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: Append to log took a long time: real 0.261s user 0.002s sys 0.000s
W20240503 14:12:57.270995 14507 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.019s sys 0.153s
W20240503 14:12:57.285336 14507 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.838s user 0.019s sys 0.154s
I20240503 14:12:57.305233 14503 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 171. Preceding OpId from leader: term: 1 index: 172. (index mismatch)
I20240503 14:12:57.306336 15281 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
I20240503 14:12:57.310809 14503 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.172: config changed from index 161 to 172, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
W20240503 14:12:57.640331 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 56028.8us
I20240503 14:12:57.727862 13010 consensus_queue.cc:237] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 178, Committed index: 178, Last appended: 1.178, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:57.736899 13269 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 178. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240503 14:12:57.738554 15281 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 178, Time since last communication: 0.000s
I20240503 14:12:57.746367 13529 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 178. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240503 14:12:57.750933 15281 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 178, Time since last communication: 0.000s
I20240503 14:12:57.773829 15361 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.179: config changed from index 172 to 179, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:57.774474 14507 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 178. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240503 14:12:57.775130 15455 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 178, Time since last communication: 0.000s
I20240503 14:12:57.776981 14502 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.179: config changed from index 172 to 179, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:57.779316 12726 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a324330d54ff4a01982e13d0cc191296 with cas_config_opid_index 172: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:57.780177 13534 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.179: config changed from index 172 to 179, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:57.781189 12737 catalog_manager.cc:5484] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: config changed from index 172 to 179, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:57.782864 13273 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.179: config changed from index 172 to 179, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:57.786927 13137 raft_consensus.cc:1261] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 178. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240503 14:12:57.787588 15361 consensus_queue.cc:1035] T a324330d54ff4a01982e13d0cc191296 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 178, Time since last communication: 0.000s
W20240503 14:12:57.810653 15525 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.052s sys 0.131s
W20240503 14:12:57.811084 15525 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.670s user 0.052s sys 0.131s
I20240503 14:12:57.792304 13144 raft_consensus.cc:2932] T a324330d54ff4a01982e13d0cc191296 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.179: config changed from index 172 to 179, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:57.937238 12863 tablet_service.cc:1514] Processing DeleteTablet for tablet a324330d54ff4a01982e13d0cc191296 with delete_type TABLET_DATA_TOMBSTONED (TS 72b07e9b53b9471989185277045db8f7 not found in new config with opid_index 179) from {username='slave'} at 127.0.0.1:43658
I20240503 14:12:57.968415 15571 tablet_replica.cc:330] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7: stopping tablet replica
I20240503 14:12:57.969089 15571 raft_consensus.cc:2229] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:58.000061 15571 raft_consensus.cc:2258] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:12:58.019091 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 53682.4us
I20240503 14:12:58.025173 15571 ts_tablet_manager.cc:1875] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:58.057682 15571 ts_tablet_manager.cc:1888] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.178
I20240503 14:12:58.060181 15571 log.cc:1196] T a324330d54ff4a01982e13d0cc191296 P 72b07e9b53b9471989185277045db8f7: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/a324330d54ff4a01982e13d0cc191296
I20240503 14:12:58.061954 12726 catalog_manager.cc:4830] TS 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): tablet a324330d54ff4a01982e13d0cc191296 (table rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]) successfully deleted
I20240503 14:12:58.234529 15186 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=185 overwritten=0 applied=185 ignored=0} inserts{seen=857 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:58.235118 15186 tablet_bootstrap.cc:492] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:12:58.235503 15186 ts_tablet_manager.cc:1367] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 4.666s user 0.313s sys 0.048s
I20240503 14:12:58.237882 15186 raft_consensus.cc:348] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:58.239766 15186 raft_consensus.cc:729] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: LEARNER
I20240503 14:12:58.240124 15186 consensus_queue.cc:260] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 185, Last appended: 1.185, Last appended by leader: 185, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:58.249982 15186 ts_tablet_manager.cc:1398] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240503 14:12:58.257458 13034 tablet_copy_service.cc:342] P 3678714622584fa4883e4bf07fcabd75: Request end of tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-37542e9a30324860bb575ae02b3c92b9 received from {username='slave'} at 127.10.145.7:47587
I20240503 14:12:58.257632 13034 tablet_copy_service.cc:433] P 3678714622584fa4883e4bf07fcabd75: ending tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-37542e9a30324860bb575ae02b3c92b9 on tablet 37542e9a30324860bb575ae02b3c92b9 with peer 21d8b64a50ba4c919cb9e711d18fb7de
W20240503 14:12:58.327792 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 105843us
I20240503 14:12:58.574246 14502 raft_consensus.cc:1203] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.182->[1.183-1.232] Dedup: 1.185->[1.186-1.232]
W20240503 14:12:58.895347 13339 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.10.145.62:35023, user_credentials={real_user=slave}} blocked reactor thread for 177969us
W20240503 14:12:58.998201 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 55667.4us
W20240503 14:12:59.110620 13078 consensus_peers.cc:485] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:59.121369 15455 raft_consensus.cc:1050] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75: attempting to promote NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de to VOTER
I20240503 14:12:59.122393 15455 consensus_queue.cc:237] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 1.237, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:59.126500 13144 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 14:12:59.127326 15455 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 236, Time since last communication: 0.001s
I20240503 14:12:59.128161 14503 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 14:12:59.129262 13531 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 14:12:59.133848 15361 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 235, Time since last communication: 0.000s
I20240503 14:12:59.170342 13404 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 14:12:59.182087 15504 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 236, Time since last communication: 0.020s
I20240503 14:12:59.187990 15504 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 236, Time since last communication: 0.000s
I20240503 14:12:59.189340 13400 raft_consensus.cc:1203] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.236->[1.237-1.238] Dedup: 1.237->[1.238-1.238]
I20240503 14:12:59.193950 15361 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.238: config changed from index 183 to 238, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:59.212325 12735 catalog_manager.cc:5484] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: config changed from index 183 to 238, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:59.215894 13143 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 183 to 238, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:59.224678 14502 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 183 to 238, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:59.223810 13529 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 183 to 238, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:59.236025 13399 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 183 to 238, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:59.324764 12882 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 14:12:59.346408 15455 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 234, Time since last communication: 0.000s
I20240503 14:12:59.418336 13534 raft_consensus.cc:1203] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.236->[1.237-1.237] Dedup: 1.237->[]
I20240503 14:12:59.537819 12877 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 183 to 238, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:59.539657 15525 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=256 overwritten=0 applied=255 ignored=0} inserts{seen=940 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:59.541812 15525 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:12:59.542672 15525 ts_tablet_manager.cc:1367] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 2.746s user 0.409s sys 0.150s
W20240503 14:12:59.547713 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 51076.9us
I20240503 14:12:59.553874 15525 raft_consensus.cc:348] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:59.556800 15525 raft_consensus.cc:729] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: LEARNER
I20240503 14:12:59.558571 15525 consensus_queue.cc:260] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 255, Last appended: 1.256, Last appended by leader: 256, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:12:59.560751 15525 ts_tablet_manager.cc:1398] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20240503 14:12:59.566115 13164 tablet_copy_service.cc:342] P 7b75d8c9e35a416ba24798fde8b9def0: Request end of tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-5e130a7b478f4f91af11bef82ebf7236 received from {username='slave'} at 127.10.145.7:52389
I20240503 14:12:59.566295 13164 tablet_copy_service.cc:433] P 7b75d8c9e35a416ba24798fde8b9def0: ending tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-5e130a7b478f4f91af11bef82ebf7236 on tablet 5e130a7b478f4f91af11bef82ebf7236 with peer 21d8b64a50ba4c919cb9e711d18fb7de
I20240503 14:12:59.625855 13014 consensus_queue.cc:237] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 1.240, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:12:59.628958 13534 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20240503 14:12:59.629928 15610 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 239, Time since last communication: 0.000s
I20240503 14:12:59.672808 12876 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20240503 14:12:59.673409 15610 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 239, Time since last communication: 0.000s
I20240503 14:12:59.692543 14507 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20240503 14:12:59.716187 15455 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 239, Time since last communication: 0.000s
I20240503 14:12:59.747931 13399 raft_consensus.cc:1261] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20240503 14:12:59.761507 14507 raft_consensus.cc:1203] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.252->[1.253-1.287] Dedup: 1.256->[1.257-1.287]
I20240503 14:12:59.813275 15281 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.241: config changed from index 238 to 241, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:59.823256 13529 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 238 to 241, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:12:59.826542 12733 catalog_manager.cc:5484] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 reported cstate change: config changed from index 238 to 241, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:59.838028 13122 tablet_service.cc:1514] Processing DeleteTablet for tablet 37542e9a30324860bb575ae02b3c92b9 with delete_type TABLET_DATA_TOMBSTONED (TS 7b75d8c9e35a416ba24798fde8b9def0 not found in new config with opid_index 241) from {username='slave'} at 127.0.0.1:34920
I20240503 14:12:59.974395 12726 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37542e9a30324860bb575ae02b3c92b9 with cas_config_opid_index 238: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:59.990722 12877 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 238 to 241, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
W20240503 14:13:00.031818 13339 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 44892.6us
I20240503 14:13:00.035591 15635 tablet_replica.cc:330] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: stopping tablet replica
I20240503 14:13:00.036610 15635 raft_consensus.cc:2229] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:00.036851 15635 pending_rounds.cc:62] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: Trying to abort 1 pending ops.
I20240503 14:13:00.036963 15635 pending_rounds.cc:68] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: Aborting op as it isn't in flight: id { term: 1 index: 240 } timestamp: 7023597893944139776 op_type: WRITE_OP write_request { tablet_id: "37542e9a30324860bb575ae02b3c92b9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000fy4%\0179\020\312\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c93d8cbdcfc5e3b7" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023597893374308352 authz_token { token_data: "\010\371\343\323\261\006\"3\n\005slave\022*\n 9a6d684b9b7f47f386c102fdfe70e8c4\020\001\030\001 \001(\001" signature: "<redacted>""+\224\372\241\021\023/C[\001s\265\352\247\340\322[p(\276\013\236Z\246(+\252\350&\006\233\200j\330)\246\237=u4\355{\344\250\304s\031i\033\004M\231\321\261\206J\205h\265l@n\311\327" signing_key_seq_num: 0 } } request_id { client_id: "92542814f4454185bc6c4b89d7ab5263" seq_no: 720 first_incomplete_seq_no: 720 attempt_no: 0 }
I20240503 14:13:00.038672 15635 raft_consensus.cc:2258] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:00.041203 15635 ts_tablet_manager.cc:1875] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:00.065578 15635 ts_tablet_manager.cc:1888] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.240
I20240503 14:13:00.065747 15635 log.cc:1196] T 37542e9a30324860bb575ae02b3c92b9 P 7b75d8c9e35a416ba24798fde8b9def0: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/37542e9a30324860bb575ae02b3c92b9
I20240503 14:13:00.066680 12728 catalog_manager.cc:4830] TS 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): tablet 37542e9a30324860bb575ae02b3c92b9 (table rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]) successfully deleted
I20240503 14:13:00.099531 15610 consensus_queue.cc:1035] T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 239, Time since last communication: 0.000s
I20240503 14:13:00.106387 14501 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 238 to 241, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.111153 13399 raft_consensus.cc:2932] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 238 to 241, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.257988 15630 raft_consensus.cc:1050] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0: attempting to promote NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de to VOTER
I20240503 14:13:00.259371 15630 consensus_queue.cc:237] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 1.293, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:13:00.262063 13533 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240503 14:13:00.262714 15067 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240503 14:13:00.267719 12876 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240503 14:13:00.270224 15067 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240503 14:13:00.274669 14507 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240503 14:13:00.279069 13403 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240503 14:13:00.281067 13273 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240503 14:13:00.325896 15067 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240503 14:13:00.326371 15579 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240503 14:13:00.337738 15630 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240503 14:13:00.365429 15579 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.294: config changed from index 253 to 294, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.370687 13533 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 253 to 294, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.377118 12733 catalog_manager.cc:5484] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 253 to 294, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.377512 13273 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 253 to 294, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.385020 12876 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 253 to 294, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.406852 14504 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 253 to 294, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.414960 13403 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 253 to 294, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
W20240503 14:13:00.495143 13337 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 47320.6us
W20240503 14:13:00.520172 13339 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 62254.8us
I20240503 14:13:00.620443 13143 consensus_queue.cc:237] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 1.295, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:13:00.624392 13529 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20240503 14:13:00.625242 15630 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20240503 14:13:00.685655 12881 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20240503 14:13:00.691375 15579 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20240503 14:13:00.700569 15501 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.296: config changed from index 294 to 296, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.703810 13270 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20240503 14:13:00.706934 12728 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e130a7b478f4f91af11bef82ebf7236 with cas_config_opid_index 294: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:00.709600 14506 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20240503 14:13:00.712386 12737 catalog_manager.cc:5484] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: config changed from index 294 to 296, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:00.712827 13529 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 294 to 296, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.725469 13382 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e130a7b478f4f91af11bef82ebf7236 with delete_type TABLET_DATA_TOMBSTONED (TS e148d8a3757340ab9288ea98cabb5ad3 not found in new config with opid_index 296) from {username='slave'} at 127.0.0.1:52634
I20240503 14:13:00.726233 15501 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20240503 14:13:00.728003 15501 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.001s
I20240503 14:13:00.734619 12877 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 294 to 296, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.748160 13269 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 294 to 296, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.756589 15667 tablet_replica.cc:330] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3: stopping tablet replica
I20240503 14:13:00.756971 15667 raft_consensus.cc:2229] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:00.757490 15667 raft_consensus.cc:2258] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:00.770018 15667 ts_tablet_manager.cc:1875] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:00.814926 14507 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 294 to 296, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:00.838287 15667 ts_tablet_manager.cc:1888] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.295
I20240503 14:13:00.838557 15667 log.cc:1196] T 5e130a7b478f4f91af11bef82ebf7236 P e148d8a3757340ab9288ea98cabb5ad3: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/wals/5e130a7b478f4f91af11bef82ebf7236
I20240503 14:13:00.846875 12727 catalog_manager.cc:4830] TS e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): tablet 5e130a7b478f4f91af11bef82ebf7236 (table rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]) successfully deleted
I20240503 14:13:00.902530 13271 consensus_queue.cc:237] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 1.241, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:13:00.907773 13013 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch)
I20240503 14:13:00.908012 12876 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch)
I20240503 14:13:00.908586 15374 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20240503 14:13:00.909132 15374 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20240503 14:13:00.915699 15638 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.242: config changed from index -1 to 242, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:13:00.918718 13143 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch)
I20240503 14:13:00.922534 13013 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index -1 to 242, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:13:00.923171 12882 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index -1 to 242, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:13:00.930881 15497 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 240, Time since last communication: 0.000s
I20240503 14:13:00.930351 12737 catalog_manager.cc:5484] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 reported cstate change: config changed from index -1 to 242, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:13:00.936887 13398 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 242. (index mismatch)
I20240503 14:13:00.964169 13143 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index -1 to 242, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:13:00.991386 15497 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 238, Time since last communication: 0.000s
I20240503 14:13:00.999497 13401 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index -1 to 242, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
W20240503 14:13:01.409608 13208 consensus_peers.cc:485] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c43a1cb5209c4a69a95877c842460adc. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:01.448115 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 45353.6us
W20240503 14:13:01.490878 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 88309.9us
I20240503 14:13:01.525475 13270 consensus_queue.cc:237] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 1.338, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:13:01.535787 13013 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 338. Preceding OpId from leader: term: 1 index: 339. (index mismatch)
I20240503 14:13:01.540575 13138 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 338. Preceding OpId from leader: term: 1 index: 339. (index mismatch)
I20240503 14:13:01.541131 15638 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I20240503 14:13:01.542716 15566 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I20240503 14:13:01.548053 13534 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 338. Preceding OpId from leader: term: 1 index: 339. (index mismatch)
I20240503 14:13:01.550350 15497 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
W20240503 14:13:01.560346 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 153870us
I20240503 14:13:01.561889 15566 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.339: config changed from index -1 to 339, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:13:01.568608 13138 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.339: config changed from index -1 to 339, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:13:01.577656 13009 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.339: config changed from index -1 to 339, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:13:01.606602 12737 catalog_manager.cc:5484] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index -1 to 339, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:01.645061 13529 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.339: config changed from index -1 to 339, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
W20240503 14:13:01.678150 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 57095.2us
I20240503 14:13:01.680779 12877 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 338. Preceding OpId from leader: term: 1 index: 339. (index mismatch)
I20240503 14:13:01.734637 15638 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 337, Time since last communication: 0.000s
W20240503 14:13:01.762845 13208 consensus_peers.cc:485] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 -> Peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Couldn't send request to peer 21d8b64a50ba4c919cb9e711d18fb7de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32029fdaf54f4bccac238b11019ec913. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:01.784191 15708 ts_tablet_manager.cc:902] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de: Initiating tablet copy from peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:13:01.857522 15708 tablet_copy_client.cc:308] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:42597
W20240503 14:13:01.871212 13467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 141702us
I20240503 14:13:01.936604 13294 tablet_copy_service.cc:140] P 2cd2acf7d8274b5a95482dffd0823a08: Received BeginTabletCopySession request for tablet c43a1cb5209c4a69a95877c842460adc from peer 21d8b64a50ba4c919cb9e711d18fb7de ({username='slave'} at 127.10.145.7:42879)
I20240503 14:13:01.936854 13294 tablet_copy_service.cc:161] P 2cd2acf7d8274b5a95482dffd0823a08: Beginning new tablet copy session on tablet c43a1cb5209c4a69a95877c842460adc from peer 21d8b64a50ba4c919cb9e711d18fb7de at {username='slave'} at 127.10.145.7:42879: session id = 21d8b64a50ba4c919cb9e711d18fb7de-c43a1cb5209c4a69a95877c842460adc
I20240503 14:13:01.939358 12881 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.339: config changed from index -1 to 339, NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) added. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } } }
I20240503 14:13:01.941119 13294 tablet_copy_source_session.cc:205] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:01.973194 15708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43a1cb5209c4a69a95877c842460adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.980211 15708 tablet_copy_client.cc:783] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:01.980470 15708 tablet_copy_client.cc:647] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:01.984556 15708 tablet_copy_client.cc:523] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:01.988386 15708 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
I20240503 14:13:02.055141 15722 ts_tablet_manager.cc:902] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de: Initiating tablet copy from peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
W20240503 14:13:02.121984 13210 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 183811us
I20240503 14:13:02.157385 15722 tablet_copy_client.cc:308] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:42597
I20240503 14:13:02.158063 13294 tablet_copy_service.cc:140] P 2cd2acf7d8274b5a95482dffd0823a08: Received BeginTabletCopySession request for tablet 32029fdaf54f4bccac238b11019ec913 from peer 21d8b64a50ba4c919cb9e711d18fb7de ({username='slave'} at 127.10.145.7:42879)
I20240503 14:13:02.158246 13294 tablet_copy_service.cc:161] P 2cd2acf7d8274b5a95482dffd0823a08: Beginning new tablet copy session on tablet 32029fdaf54f4bccac238b11019ec913 from peer 21d8b64a50ba4c919cb9e711d18fb7de at {username='slave'} at 127.10.145.7:42879: session id = 21d8b64a50ba4c919cb9e711d18fb7de-32029fdaf54f4bccac238b11019ec913
I20240503 14:13:02.161880 13294 tablet_copy_source_session.cc:205] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:02.164778 15722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32029fdaf54f4bccac238b11019ec913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.197147 15722 tablet_copy_client.cc:783] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:02.199507 15722 tablet_copy_client.cc:647] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:02.208734 15722 tablet_copy_client.cc:523] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:02.214689 15722 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap starting.
W20240503 14:13:02.399561 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 120388us
W20240503 14:13:02.729411 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 264926us
W20240503 14:13:03.061944 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 44279.3us
W20240503 14:13:03.114228 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 45947.2us
I20240503 14:13:03.140074 15708 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=245 overwritten=0 applied=244 ignored=0} inserts{seen=929 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:13:03.140548 15708 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:13:03.141233 15708 ts_tablet_manager.cc:1367] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 1.153s user 0.291s sys 0.030s
I20240503 14:13:03.142562 15708 raft_consensus.cc:348] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:13:03.143751 15708 raft_consensus.cc:729] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: LEARNER
I20240503 14:13:03.143990 15708 consensus_queue.cc:260] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 244, Last appended: 1.245, Last appended by leader: 245, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:13:03.150771 15708 ts_tablet_manager.cc:1398] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240503 14:13:03.154484 13294 tablet_copy_service.cc:342] P 2cd2acf7d8274b5a95482dffd0823a08: Request end of tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-c43a1cb5209c4a69a95877c842460adc received from {username='slave'} at 127.10.145.7:42879
I20240503 14:13:03.154654 13294 tablet_copy_service.cc:433] P 2cd2acf7d8274b5a95482dffd0823a08: ending tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-c43a1cb5209c4a69a95877c842460adc on tablet c43a1cb5209c4a69a95877c842460adc with peer 21d8b64a50ba4c919cb9e711d18fb7de
I20240503 14:13:03.246438 15722 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=340 overwritten=0 applied=340 ignored=0} inserts{seen=1025 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:03.246863 15722 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de: Bootstrap complete.
I20240503 14:13:03.247184 15722 ts_tablet_manager.cc:1367] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent bootstrapping tablet: real 1.033s user 0.376s sys 0.029s
I20240503 14:13:03.247992 15722 raft_consensus.cc:348] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:13:03.248289 15722 raft_consensus.cc:729] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21d8b64a50ba4c919cb9e711d18fb7de, State: Initialized, Role: LEARNER
I20240503 14:13:03.248497 15722 consensus_queue.cc:260] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 340, Last appended: 1.340, Last appended by leader: 340, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: true } }
I20240503 14:13:03.249085 15722 ts_tablet_manager.cc:1398] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:03.250835 13294 tablet_copy_service.cc:342] P 2cd2acf7d8274b5a95482dffd0823a08: Request end of tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-32029fdaf54f4bccac238b11019ec913 received from {username='slave'} at 127.10.145.7:42879
I20240503 14:13:03.250962 13294 tablet_copy_service.cc:433] P 2cd2acf7d8274b5a95482dffd0823a08: ending tablet copy session 21d8b64a50ba4c919cb9e711d18fb7de-32029fdaf54f4bccac238b11019ec913 on tablet 32029fdaf54f4bccac238b11019ec913 with peer 21d8b64a50ba4c919cb9e711d18fb7de
I20240503 14:13:03.378010 14503 raft_consensus.cc:1203] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.338->[1.339-1.351] Dedup: 1.340->[1.341-1.351]
I20240503 14:13:03.446120 15721 raft_consensus.cc:1050] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08: attempting to promote NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de to VOTER
I20240503 14:13:03.449468 15721 consensus_queue.cc:237] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 1.352, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:13:03.458241 13534 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20240503 14:13:03.459380 15374 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 14:13:03.460287 14503 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20240503 14:13:03.460829 15497 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20240503 14:13:03.496722 12879 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20240503 14:13:03.498944 15497 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20240503 14:13:03.505131 15374 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.353: config changed from index 339 to 353, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.511896 12875 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.353: config changed from index 339 to 353, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.519692 13138 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20240503 14:13:03.520692 15497 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 14:13:03.522884 13138 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.353: config changed from index 339 to 353, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.534929 12734 catalog_manager.cc:5484] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 reported cstate change: config changed from index 339 to 353, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.539126 13011 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20240503 14:13:03.542523 15729 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 351, Time since last communication: 0.000s
I20240503 14:13:03.543659 14507 raft_consensus.cc:1203] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.241->[1.242-1.263] Dedup: 1.245->[1.246-1.263]
W20240503 14:13:03.582144 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 43488.9us
W20240503 14:13:03.609195 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 58479.8us
I20240503 14:13:03.626817 14504 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.353: config changed from index 339 to 353, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.638598 13533 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.353: config changed from index 339 to 353, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.667181 13009 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.353: config changed from index 339 to 353, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.705039 15737 raft_consensus.cc:1050] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08: attempting to promote NON_VOTER 21d8b64a50ba4c919cb9e711d18fb7de to VOTER
I20240503 14:13:03.709970 15737 consensus_queue.cc:237] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 1.263, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:13:03.714612 13137 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20240503 14:13:03.715148 15729 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 14:13:03.718006 14504 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 LEARNER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20240503 14:13:03.718515 15497 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 14:13:03.720805 12875 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
W20240503 14:13:03.721719 12947 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 45150.1us
I20240503 14:13:03.733230 13013 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20240503 14:13:03.734114 15729 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 14:13:03.734319 13401 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20240503 14:13:03.735169 15729 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 14:13:03.748514 15729 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20240503 14:13:03.749234 15721 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.264: config changed from index 242 to 264, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.754616 13143 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 242 to 264, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.757062 14504 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 242 to 264, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.762081 13401 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 242 to 264, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.784973 12739 catalog_manager.cc:5484] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 reported cstate change: config changed from index 242 to 264, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.803292 13009 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 242 to 264, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.804795 12877 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 242 to 264, 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.860813 13271 consensus_queue.cc:237] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 1.357, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:13:03.861189 13273 consensus_queue.cc:237] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 1.266, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:13:03.864651 13143 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240503 14:13:03.864714 13013 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20240503 14:13:03.865233 15497 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 356, Time since last communication: 0.000s
I20240503 14:13:03.865187 15800 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 265, Time since last communication: 0.000s
I20240503 14:13:03.866662 12880 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240503 14:13:03.866672 14501 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 265. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240503 14:13:03.866767 13403 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240503 14:13:03.866842 14502 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20240503 14:13:03.867735 15737 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20240503 14:13:03.868553 15729 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 265, Time since last communication: 0.000s
I20240503 14:13:03.869488 15737 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 265, Time since last communication: 0.000s
I20240503 14:13:03.870239 15497 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 265, Time since last communication: 0.000s
I20240503 14:13:03.873939 15374 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.358: config changed from index 353 to 358, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.875195 15721 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.267: config changed from index 264 to 267, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.878651 12727 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32029fdaf54f4bccac238b11019ec913 with cas_config_opid_index 353: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:03.878970 12727 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c43a1cb5209c4a69a95877c842460adc with cas_config_opid_index 264: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:03.880331 13144 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20240503 14:13:03.880542 12737 catalog_manager.cc:5484] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index 353 to 358, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:03.883085 15800 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20240503 14:13:03.882894 13011 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 353 to 358, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
W20240503 14:13:03.885025 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 44177.1us
I20240503 14:13:03.885224 13143 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 353 to 358, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.887737 14507 raft_consensus.cc:1203] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.265->[1.266-1.266] Dedup: 1.266->[]
I20240503 14:13:03.890666 14501 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 264 to 267, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.891067 13399 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 264 to 267, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.896317 12863 tablet_service.cc:1514] Processing DeleteTablet for tablet 32029fdaf54f4bccac238b11019ec913 with delete_type TABLET_DATA_TOMBSTONED (TS 72b07e9b53b9471989185277045db8f7 not found in new config with opid_index 358) from {username='slave'} at 127.0.0.1:43658
I20240503 14:13:03.897876 12742 catalog_manager.cc:5484] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 reported cstate change: config changed from index 264 to 267, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.902633 13144 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 264 to 267, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.962558 12877 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 264 to 267, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.949178 14507 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 353 to 358, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:03.988613 13534 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20240503 14:13:04.002755 15800 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 356, Time since last communication: 0.000s
I20240503 14:13:04.003063 12992 tablet_service.cc:1514] Processing DeleteTablet for tablet c43a1cb5209c4a69a95877c842460adc with delete_type TABLET_DATA_TOMBSTONED (TS 3678714622584fa4883e4bf07fcabd75 not found in new config with opid_index 267) from {username='slave'} at 127.0.0.1:55928
I20240503 14:13:04.042842 15805 tablet_replica.cc:330] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75: stopping tablet replica
I20240503 14:13:04.043156 15805 raft_consensus.cc:2229] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:04.044027 15805 raft_consensus.cc:2258] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:04.045228 15805 ts_tablet_manager.cc:1875] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:04.051180 13529 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 353 to 358, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } }
I20240503 14:13:04.074468 15804 tablet_replica.cc:330] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: stopping tablet replica
I20240503 14:13:04.074817 15804 raft_consensus.cc:2229] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:04.101847 15804 pending_rounds.cc:62] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: Trying to abort 1 pending ops.
I20240503 14:13:04.102124 15804 pending_rounds.cc:68] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: Aborting op as it isn't in flight: id { term: 1 index: 357 } timestamp: 7023597911381016576 op_type: WRITE_OP write_request { tablet_id: "32029fdaf54f4bccac238b11019ec913" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\316Io)Ki$P\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""b8af664586c6f3d3" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023597911143456768 authz_token { token_data: "\010\371\343\323\261\006\"3\n\005slave\022*\n 0ddf2a368d7344a8a2eeeaf44782ed5c\020\001\030\001 \001(\001" signature: "<redacted>""ec\331\271\365\312\030\350\222\\\222p\017\337\263\027\353\202\330A\032\231&\022\304]7\241\022k\262\341[\342\001\001\305\256\263U]\237\017\227XN\316\250\020\311\377om\330N0\271W;8A-\307\353" signing_key_seq_num: 0 } } request_id { client_id: "3a85788554514deca978e3ff14977416" seq_no: 1037 first_incomplete_seq_no: 1037 attempt_no: 0 }
I20240503 14:13:04.103263 15804 raft_consensus.cc:2258] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:04.110500 15804 ts_tablet_manager.cc:1875] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:04.153179 15804 ts_tablet_manager.cc:1888] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.357
I20240503 14:13:04.177654 15804 log.cc:1196] T 32029fdaf54f4bccac238b11019ec913 P 72b07e9b53b9471989185277045db8f7: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/32029fdaf54f4bccac238b11019ec913
I20240503 14:13:04.187348 12726 catalog_manager.cc:4830] TS 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): tablet 32029fdaf54f4bccac238b11019ec913 (table rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]) successfully deleted
I20240503 14:13:04.288214 15805 ts_tablet_manager.cc:1888] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.265
I20240503 14:13:04.288362 15805 log.cc:1196] T c43a1cb5209c4a69a95877c842460adc P 3678714622584fa4883e4bf07fcabd75: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/c43a1cb5209c4a69a95877c842460adc
I20240503 14:13:04.354305 12726 catalog_manager.cc:4830] TS 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): tablet c43a1cb5209c4a69a95877c842460adc (table rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]) successfully deleted
W20240503 14:13:04.484397 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 120703us
W20240503 14:13:05.212482 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 60293.2us
W20240503 14:13:05.894845 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 50858.3us
W20240503 14:13:06.644023 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 123049us
W20240503 14:13:07.179708 12816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 119242us
W20240503 14:13:07.179699 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 120324us
W20240503 14:13:07.297113 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 72349.2us
W20240503 14:13:07.539898 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 44721.6us
W20240503 14:13:07.686574 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 67463us
W20240503 14:13:07.836616 12816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 43975.4us
I20240503 14:13:08.103705 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:13:08.107478 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 59178.2us
W20240503 14:13:08.110936 12950 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 50348us
I20240503 14:13:08.185423 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:08.205328 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:08.275100 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:08.330385 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:08.465795 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:08.513559 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:13:08.588861 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 45250.5us
W20240503 14:13:08.797619 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 45087.9us
W20240503 14:13:09.237159 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 82171.4us
W20240503 14:13:09.254878 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 147751us
W20240503 14:13:09.254946 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 128159us
W20240503 14:13:09.254906 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 145560us
W20240503 14:13:09.383523 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 50835.2us
I20240503 14:13:09.523579 13531 consensus_queue.cc:237] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 373, Committed index: 373, Last appended: 1.373, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: true } }
W20240503 14:13:09.531677 13467 consensus_peers.cc:485] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 -> Peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Couldn't send request to peer 3678714622584fa4883e4bf07fcabd75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71289cddae864085beacce5aa1fb0c4e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:09.533106 12880 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20240503 14:13:09.533766 15889 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20240503 14:13:09.536903 14507 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20240503 14:13:09.538833 15889 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20240503 14:13:09.541841 15419 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.374: config changed from index 217 to 374, NON_VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: true } } }
I20240503 14:13:09.547509 13404 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20240503 14:13:09.548058 12881 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 217 to 374, NON_VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: true } } }
I20240503 14:13:09.549763 12742 catalog_manager.cc:5484] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 217 to 374, NON_VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) added. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:09.551558 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 44036.3us
I20240503 14:13:09.553342 13143 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20240503 14:13:09.559541 14507 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 217 to 374, NON_VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: true } } }
I20240503 14:13:09.560834 15889 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20240503 14:13:09.576128 13404 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 217 to 374, NON_VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: true } } }
I20240503 14:13:09.624400 15419 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 372, Time since last communication: 0.000s
I20240503 14:13:09.640427 13143 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 217 to 374, NON_VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: true } } }
W20240503 14:13:09.699728 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 45536.3us
I20240503 14:13:10.247946 15973 ts_tablet_manager.cc:902] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: Initiating tablet copy from peer af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
W20240503 14:13:10.393070 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 109839us
W20240503 14:13:10.691711 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 66566.9us
W20240503 14:13:11.136219 15973 thread.cc:636] tablet-download-pool-71289cddae864085beacce5aa1fb0c4e [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.041s sys 0.083s
W20240503 14:13:11.136435 15973 thread.cc:602] tablet-download-pool-71289cddae864085beacce5aa1fb0c4e [worker] (thread pool) Time spent starting thread: real 0.750s user 0.041s sys 0.083s
I20240503 14:13:11.144129 15973 tablet_copy_client.cc:308] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36377
I20240503 14:13:11.150816 13554 tablet_copy_service.cc:140] P af88d71da3a54f779cf4f651f4393bb6: Received BeginTabletCopySession request for tablet 71289cddae864085beacce5aa1fb0c4e from peer 3678714622584fa4883e4bf07fcabd75 ({username='slave'} at 127.10.145.2:41435)
I20240503 14:13:11.151005 13554 tablet_copy_service.cc:161] P af88d71da3a54f779cf4f651f4393bb6: Beginning new tablet copy session on tablet 71289cddae864085beacce5aa1fb0c4e from peer 3678714622584fa4883e4bf07fcabd75 at {username='slave'} at 127.10.145.2:41435: session id = 3678714622584fa4883e4bf07fcabd75-71289cddae864085beacce5aa1fb0c4e
I20240503 14:13:11.158020 13554 tablet_copy_source_session.cc:205] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:11.167440 15973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71289cddae864085beacce5aa1fb0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:11.202262 15973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.10.145.6:58535 (request call id 483) took 1256 ms. Trace:
I20240503 14:13:11.202659 15973 rpcz_store.cc:276] 0503 14:13:09.946048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:13:09.960650 (+ 14602us) service_pool.cc:225] Handling call
0503 14:13:10.248841 (+288191us) ts_tablet_manager.cc:903] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: Initiating tablet copy from peer af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
0503 14:13:11.164351 (+915510us) tablet_copy_client.cc:353] Tablet copy session begun
0503 14:13:11.179858 (+ 15507us) tablet_metadata.cc:641] Metadata flushed
0503 14:13:11.180007 (+ 149us) tablet_copy_client.cc:482] Wrote new tablet metadata
0503 14:13:11.198422 (+ 18415us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 14:13:11.202031 (+ 3609us) tablet_replica.cc:205] Creating consensus instance
0503 14:13:11.202189 (+ 158us) ts_tablet_manager.cc:958] Replica registered.
0503 14:13:11.202227 (+ 38us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":108,"tablet-copy.queue_time_us":286700,"thread_start_us":1183302,"threads_started":5}
I20240503 14:13:11.213862 15973 tablet_copy_client.cc:783] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:11.214376 15973 tablet_copy_client.cc:647] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:11.221025 15973 tablet_copy_client.cc:523] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:11.238652 15973 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: Bootstrap starting.
W20240503 14:13:11.243304 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 89146.9us
W20240503 14:13:11.558346 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 53875.9us
W20240503 14:13:12.090497 12819 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 93417.4us
W20240503 14:13:12.980312 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 64522.9us
W20240503 14:13:13.387219 13467 consensus_peers.cc:485] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 -> Peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Couldn't send request to peer 3678714622584fa4883e4bf07fcabd75. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:13.709364 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 46471.1us
W20240503 14:13:13.735723 13339 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 53482.3us
W20240503 14:13:13.893447 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 113851us
W20240503 14:13:14.118685 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 54436.2us
W20240503 14:13:14.794337 13077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 203592us
W20240503 14:13:15.425948 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 55751.8us
W20240503 14:13:15.429433 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 68042.4us
W20240503 14:13:15.959080 13467 consensus_peers.cc:485] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 -> Peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Couldn't send request to peer 3678714622584fa4883e4bf07fcabd75. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
I20240503 14:13:16.333012 15973 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: Bootstrap replayed 1/1 log segments. Stats: ops{read=391 overwritten=0 applied=391 ignored=0} inserts{seen=1019 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:16.334956 15973 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: Bootstrap complete.
I20240503 14:13:16.337836 15973 ts_tablet_manager.cc:1367] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: Time spent bootstrapping tablet: real 5.099s user 0.504s sys 0.029s
I20240503 14:13:16.340001 15973 raft_consensus.cc:348] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: true } }
I20240503 14:13:16.340462 15973 raft_consensus.cc:729] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3678714622584fa4883e4bf07fcabd75, State: Initialized, Role: LEARNER
I20240503 14:13:16.340806 15973 consensus_queue.cc:260] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 391, Last appended: 1.391, Last appended by leader: 391, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: true } }
I20240503 14:13:16.388375 15973 ts_tablet_manager.cc:1398] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20240503 14:13:16.446614 13014 raft_consensus.cc:1203] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.373->[1.374-1.433] Dedup: 1.391->[1.392-1.433]
W20240503 14:13:16.511512 12816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 63454.2us
W20240503 14:13:16.513190 12819 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 58881.8us
W20240503 14:13:16.515026 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 74160.6us
I20240503 14:13:16.592533 13554 tablet_copy_service.cc:342] P af88d71da3a54f779cf4f651f4393bb6: Request end of tablet copy session 3678714622584fa4883e4bf07fcabd75-71289cddae864085beacce5aa1fb0c4e received from {username='slave'} at 127.10.145.2:41435
I20240503 14:13:16.592716 13554 tablet_copy_service.cc:433] P af88d71da3a54f779cf4f651f4393bb6: ending tablet copy session 3678714622584fa4883e4bf07fcabd75-71289cddae864085beacce5aa1fb0c4e on tablet 71289cddae864085beacce5aa1fb0c4e with peer 3678714622584fa4883e4bf07fcabd75
W20240503 14:13:16.684288 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 121391us
I20240503 14:13:16.735378 16114 raft_consensus.cc:1050] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: attempting to promote NON_VOTER 3678714622584fa4883e4bf07fcabd75 to VOTER
I20240503 14:13:16.736727 16114 consensus_queue.cc:237] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 434, Committed index: 434, Last appended: 1.434, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } }
I20240503 14:13:16.746963 13401 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 1 index: 435. (index mismatch)
I20240503 14:13:16.753392 16012 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I20240503 14:13:16.758988 13011 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 LEARNER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 1 index: 435. (index mismatch)
I20240503 14:13:16.760501 16012 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I20240503 14:13:16.768848 12875 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 1 index: 435. (index mismatch)
I20240503 14:13:16.791134 16012 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I20240503 14:13:16.797273 14503 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 1 index: 435. (index mismatch)
I20240503 14:13:16.803320 16042 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I20240503 14:13:16.822835 16095 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.435: config changed from index 374 to 435, 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
I20240503 14:13:16.829589 12880 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.435: config changed from index 374 to 435, 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
I20240503 14:13:16.838770 12742 catalog_manager.cc:5484] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 reported cstate change: config changed from index 374 to 435, 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
I20240503 14:13:16.844422 13014 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.435: config changed from index 374 to 435, 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
I20240503 14:13:16.942131 13143 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 1 index: 435. (index mismatch)
I20240503 14:13:16.942241 13404 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.435: config changed from index 374 to 435, 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
I20240503 14:13:16.943521 16012 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.001s
I20240503 14:13:16.951756 14503 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.435: config changed from index 374 to 435, 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
I20240503 14:13:16.957845 13142 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.435: config changed from index 374 to 435, 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
W20240503 14:13:17.012084 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 91220.7us
W20240503 14:13:17.058213 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 94624.8us
I20240503 14:13:17.712750 13534 consensus_queue.cc:237] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 435, Committed index: 435, Last appended: 1.436, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 437 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } }
I20240503 14:13:17.717803 13143 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 436. Preceding OpId from leader: term: 1 index: 437. (index mismatch)
I20240503 14:13:17.718402 13011 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 436. Preceding OpId from leader: term: 1 index: 437. (index mismatch)
I20240503 14:13:17.718513 16146 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 437, Last known committed idx: 436, Time since last communication: 0.000s
I20240503 14:13:17.718838 14502 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 436. Preceding OpId from leader: term: 1 index: 437. (index mismatch)
I20240503 14:13:17.720269 16146 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 437, Last known committed idx: 436, Time since last communication: 0.000s
I20240503 14:13:17.720572 16152 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 437, Last known committed idx: 436, Time since last communication: 0.000s
I20240503 14:13:17.725198 14502 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.437: config changed from index 435 to 437, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 437 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
I20240503 14:13:17.743031 12733 catalog_manager.cc:5484] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de reported cstate change: config changed from index 435 to 437, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 437 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
I20240503 14:13:17.780092 12863 tablet_service.cc:1514] Processing DeleteTablet for tablet 71289cddae864085beacce5aa1fb0c4e with delete_type TABLET_DATA_TOMBSTONED (TS 72b07e9b53b9471989185277045db8f7 not found in new config with opid_index 437) from {username='slave'} at 127.0.0.1:43658
I20240503 14:13:17.801817 13404 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 436. Preceding OpId from leader: term: 1 index: 437. (index mismatch)
I20240503 14:13:17.802434 16152 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 437, Last known committed idx: 436, Time since last communication: 0.000s
I20240503 14:13:17.803927 13401 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.437: config changed from index 435 to 437, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 437 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
W20240503 14:13:17.818324 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 110159us
I20240503 14:13:17.828250 16189 tablet_replica.cc:330] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: stopping tablet replica
I20240503 14:13:17.828717 16189 raft_consensus.cc:2229] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:17.829138 16189 pending_rounds.cc:62] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: Trying to abort 1 pending ops.
I20240503 14:13:17.829449 16189 pending_rounds.cc:68] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: Aborting op as it isn't in flight: id { term: 1 index: 436 } timestamp: 7023597967339024384 op_type: WRITE_OP write_request { tablet_id: "71289cddae864085beacce5aa1fb0c4e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\202\353\376J\214\213\340\343\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""ad1628d213a4143c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023597967224352768 authz_token { token_data: "\010\371\343\323\261\006\"3\n\005slave\022*\n efd4bd07f51242c7a453d29157fb1cc1\020\001\030\001 \001(\001" signature: "<redacted>""\027<6\007\227\004o\022K?\244\254\344\'];\331\336\245q\342B\242v\260\227}\352\035\233\366Hf]\341\246\023l<\027QH\373\276O\264\302\277\310\270\360\265t\262\377\205\206\027\233f\213\377\216&" signing_key_seq_num: 0 } } request_id { client_id: "aaa6899694b747809c724488d5af3f93" seq_no: 1264 first_incomplete_seq_no: 1264 attempt_no: 0 }
I20240503 14:13:17.830909 16189 raft_consensus.cc:2258] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:17.830876 13010 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.437: config changed from index 435 to 437, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 437 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
I20240503 14:13:17.836419 16095 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.437: config changed from index 435 to 437, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 437 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
I20240503 14:13:17.841406 12729 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71289cddae864085beacce5aa1fb0c4e with cas_config_opid_index 435: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 14:13:17.990165 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 93914.4us
I20240503 14:13:18.018945 13138 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.437: config changed from index 435 to 437, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 437 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } }
W20240503 14:13:18.051733 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 232967us
I20240503 14:13:18.239178 16189 ts_tablet_manager.cc:1875] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:18.380671 16189 ts_tablet_manager.cc:1888] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.436
I20240503 14:13:18.406973 16189 log.cc:1196] T 71289cddae864085beacce5aa1fb0c4e P 72b07e9b53b9471989185277045db8f7: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/71289cddae864085beacce5aa1fb0c4e
I20240503 14:13:18.408087 12726 catalog_manager.cc:4830] TS 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): tablet 71289cddae864085beacce5aa1fb0c4e (table rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]) successfully deleted
W20240503 14:13:18.683681 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 73494.2us
W20240503 14:13:18.843032 12819 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 179960us
W20240503 14:13:19.406397 12947 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 95663.8us
W20240503 14:13:19.650851 12878 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.005s sys 0.012s
W20240503 14:13:19.693701 12878 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s user 0.005s sys 0.012s
W20240503 14:13:19.982555 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 47589.8us
W20240503 14:13:20.020614 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 65139.7us
I20240503 14:13:20.268528 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:13:20.301878 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 113573us
I20240503 14:13:20.303629 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:20.365260 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:20.517062 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:20.534628 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:20.542192 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:13:20.678555 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 70218.8us
I20240503 14:13:20.719419 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:13:21.159864 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 97790.6us
W20240503 14:13:21.367615 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 61034.8us
W20240503 14:13:21.867106 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 69230.4us
W20240503 14:13:22.031433 12949 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 46199.8us
W20240503 14:13:22.312574 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 101818us
W20240503 14:13:22.433183 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 56243.8us
W20240503 14:13:22.897483 13077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 57574.8us
W20240503 14:13:22.899227 13080 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 57877.9us
W20240503 14:13:22.911295 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 71872.7us
W20240503 14:13:23.192727 13337 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 226943us
I20240503 14:13:24.094079 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:24.125353 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:24.145534 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:24.151417 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:24.177147 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:24.203436 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:13:24.251633 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 84579.9us
I20240503 14:13:24.266553 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:13:24.680042 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 77736us
W20240503 14:13:24.880761 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 68503.2us
W20240503 14:13:25.206445 12950 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 79038us
W20240503 14:13:25.409883 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 189409us
W20240503 14:13:25.437986 12950 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 217738us
W20240503 14:13:25.518355 12950 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 53752.2us
W20240503 14:13:26.031415 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 54849.4us
W20240503 14:13:26.485668 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 151441us
W20240503 14:13:27.421952 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 44289.9us
W20240503 14:13:27.976809 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 67962us
W20240503 14:13:28.236140 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 44571.6us
W20240503 14:13:28.241154 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 49945.2us
W20240503 14:13:28.376462 14425 kernel_stack_watchdog.cc:198] Thread 14506 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 277ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f1d01d12980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f1cfe980d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f1cfe9a17ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f1cfe99d3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f1cfe9a24ff kudu::ThreadPool::Submit() at ??:0
@ 0x7f1d0b50ccf9 kudu::tablet::OpDriver::ApplyAsync() at ??:0
@ 0x7f1d0b50db3d kudu::tablet::OpDriver::ReplicationFinished() at ??:0
@ 0x7f1d0b4be874 kudu::tablet::TabletReplica::StartFollowerOp()::$_7::operator()() at ??:0
@ 0x7f1d0b4be6dd std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1d11eaf17e std::function<>::operator()() at ??:0
@ 0x7f1d0a1d905d kudu::consensus::ConsensusRound::NotifyReplicationFinished() at ??:0
@ 0x7f1d0a16651d kudu::consensus::PendingRounds::AdvanceCommittedIndex() at ??:0
@ 0x7f1d0a1af737 kudu::consensus::RaftConsensus::UpdateReplica() at ??:0
W20240503 14:13:28.492220 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 546252us
W20240503 14:13:28.637964 14506 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.736s user 0.075s sys 0.087s
W20240503 14:13:28.638123 14506 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.736s user 0.075s sys 0.087s
I20240503 14:13:28.853885 16481 raft_consensus.cc:482] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:13:28.854113 16481 raft_consensus.cc:504] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } }
I20240503 14:13:28.857265 13532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:13:28.858328 16481 leader_election.cc:288] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
I20240503 14:13:28.971236 13011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3678714622584fa4883e4bf07fcabd75" is_pre_election: true
W20240503 14:13:29.045192 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 73243.1us
I20240503 14:13:29.051122 14504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" is_pre_election: true
I20240503 14:13:29.054181 13400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "2cd2acf7d8274b5a95482dffd0823a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
W20240503 14:13:29.061578 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 64475.4us
W20240503 14:13:29.102224 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 49554.1us
I20240503 14:13:29.102596 13208 leader_election.cc:302] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 2cd2acf7d8274b5a95482dffd0823a08; no voters: 21d8b64a50ba4c919cb9e711d18fb7de, 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6
I20240503 14:13:29.120100 16463 raft_consensus.cc:2726] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:13:29.208302 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 60148.3us
W20240503 14:13:29.213805 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 97844.1us
W20240503 14:13:29.389063 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 183810us
W20240503 14:13:29.486994 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 45018.4us
W20240503 14:13:29.541672 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 70650.1us
W20240503 14:13:29.548552 14502 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.008s sys 0.057s
W20240503 14:13:29.548707 14502 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s user 0.008s sys 0.057s
W20240503 14:13:29.764057 16565 log.cc:924] Time spent T 37542e9a30324860bb575ae02b3c92b9 P 3678714622584fa4883e4bf07fcabd75: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20240503 14:13:29.786409 13208 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 89219.9us
W20240503 14:13:29.803086 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 131522us
W20240503 14:13:30.073702 12816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 57138.1us
W20240503 14:13:30.075131 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 61185.8us
W20240503 14:13:30.151615 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 66731us
W20240503 14:13:30.481405 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 51207.1us
W20240503 14:13:30.537782 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 45019.6us
W20240503 14:13:30.874048 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 151441us
W20240503 14:13:31.222600 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 52628.2us
I20240503 14:13:31.234438 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:31.307984 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:13:31.409106 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 76603.9us
I20240503 14:13:31.469401 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:31.493187 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:31.639881 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:13:31.663146 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 58256.1us
I20240503 14:13:31.669553 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:13:31.724812 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 56160.9us
I20240503 14:13:31.767717 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:13:31.888480 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 48313.9us
W20240503 14:13:32.358856 13338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 107097us
W20240503 14:13:32.601761 13467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 46070.8us
W20240503 14:13:32.650513 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 75658.7us
W20240503 14:13:32.651178 12949 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 70810.3us
W20240503 14:13:32.651378 12950 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36377, user_credentials={real_user=slave}} blocked reactor thread for 66898.9us
W20240503 14:13:32.672227 13078 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 89840.8us
W20240503 14:13:33.083062 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 116485us
W20240503 14:13:33.389339 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 97390.1us
W20240503 14:13:33.669185 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 57752.9us
W20240503 14:13:33.967383 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 48424.9us
I20240503 14:13:34.190918 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:13:34.258751 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 60666.5us
W20240503 14:13:34.262380 12949 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 46815.4us
I20240503 14:13:34.305704 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:34.354926 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:13:34.482326 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 57591.6us
I20240503 14:13:34.493880 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:34.538321 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:34.590855 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:34.723232 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:13:36.365000 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 74978.8us
W20240503 14:13:36.385583 13207 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:38531, user_credentials={real_user=slave}} blocked reactor thread for 66407.6us
W20240503 14:13:36.417701 13209 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 52283.9us
W20240503 14:13:36.530380 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 68873.7us
W20240503 14:13:36.705493 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 92610.2us
W20240503 14:13:36.996028 12818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 69129.4us
W20240503 14:13:36.996309 12817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 74302.1us
W20240503 14:13:36.996505 12816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 59392us
W20240503 14:13:37.288040 13469 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:41101, user_credentials={real_user=slave}} blocked reactor thread for 52781.6us
W20240503 14:13:37.289177 13467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:36441, user_credentials={real_user=slave}} blocked reactor thread for 57907.1us
W20240503 14:13:37.308579 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 109792us
W20240503 14:13:37.363130 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.4:42597, user_credentials={real_user=slave}} blocked reactor thread for 52957.6us
W20240503 14:13:37.429358 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 47865.4us
W20240503 14:13:37.700582 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.5:38005, user_credentials={real_user=slave}} blocked reactor thread for 70290.2us
W20240503 14:13:38.222956 12948 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:44891, user_credentials={real_user=slave}} blocked reactor thread for 100433us
I20240503 14:13:39.043283 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:39.064898 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:39.074082 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:39.100425 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:39.123179 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:39.240878 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:39.245335 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
587e2eb33b9846ca8ab894d34f10c964 | 127.10.145.62:35023 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.145.20:46469 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
21d8b64a50ba4c919cb9e711d18fb7de | 127.10.145.7:44891 | HEALTHY | <none> | 0 | 0
2cd2acf7d8274b5a95482dffd0823a08 | 127.10.145.4:42597 | HEALTHY | <none> | 3 | 0
3678714622584fa4883e4bf07fcabd75 | 127.10.145.2:36441 | HEALTHY | <none> | 5 | 0
72b07e9b53b9471989185277045db8f7 | 127.10.145.1:38531 | HEALTHY | <none> | 2 | 0
7b75d8c9e35a416ba24798fde8b9def0 | 127.10.145.3:41101 | HEALTHY | <none> | 2 | 0
af88d71da3a54f779cf4f651f4393bb6 | 127.10.145.6:36377 | HEALTHY | <none> | 2 | 0
e148d8a3757340ab9288ea98cabb5ad3 | 127.10.145.5:38005 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.10.145.1 | experimental | 127.10.145.1:38531
local_ip_for_outbound_sockets | 127.10.145.2 | experimental | 127.10.145.2:36441
local_ip_for_outbound_sockets | 127.10.145.3 | experimental | 127.10.145.3:41101
local_ip_for_outbound_sockets | 127.10.145.4 | experimental | 127.10.145.4:42597
local_ip_for_outbound_sockets | 127.10.145.5 | experimental | 127.10.145.5:38005
local_ip_for_outbound_sockets | 127.10.145.6 | experimental | 127.10.145.6:36377
local_ip_for_outbound_sockets | 127.10.145.7 | experimental | 127.10.145.7:44891
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.145.1:38531
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.145.2:36441
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.145.3:41101
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.10.145.4:42597
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.10.145.5:38005
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.10.145.6:36377
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.10.145.7:44891
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.145.20:46469 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 11
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:13:39.313812 10820 log_verifier.cc:126] Checking tablet 1bfebd34071946398632ab7597622a41
I20240503 14:13:39.782730 10820 log_verifier.cc:177] Verified matching terms for 468 ops in tablet 1bfebd34071946398632ab7597622a41
I20240503 14:13:39.783970 10820 log_verifier.cc:126] Checking tablet 3117228f0f7c431981bf0173b3b3344f
I20240503 14:13:40.317409 10820 log_verifier.cc:177] Verified matching terms for 576 ops in tablet 3117228f0f7c431981bf0173b3b3344f
I20240503 14:13:40.319255 10820 log_verifier.cc:126] Checking tablet 32029fdaf54f4bccac238b11019ec913
I20240503 14:13:40.988754 10820 log_verifier.cc:177] Verified matching terms for 735 ops in tablet 32029fdaf54f4bccac238b11019ec913
I20240503 14:13:40.992218 10820 log_verifier.cc:126] Checking tablet 37542e9a30324860bb575ae02b3c92b9
I20240503 14:13:41.445070 10820 log_verifier.cc:177] Verified matching terms for 476 ops in tablet 37542e9a30324860bb575ae02b3c92b9
I20240503 14:13:41.446326 10820 log_verifier.cc:126] Checking tablet 3d17c42814e24b70b25c24ea15a336f3
I20240503 14:13:41.910389 10820 log_verifier.cc:177] Verified matching terms for 503 ops in tablet 3d17c42814e24b70b25c24ea15a336f3
I20240503 14:13:41.911681 10820 log_verifier.cc:126] Checking tablet 407844cff18c4421ab36e61e58ab5ee9
I20240503 14:13:42.553438 10820 log_verifier.cc:177] Verified matching terms for 719 ops in tablet 407844cff18c4421ab36e61e58ab5ee9
I20240503 14:13:42.555480 10820 log_verifier.cc:126] Checking tablet 5e130a7b478f4f91af11bef82ebf7236
I20240503 14:13:43.176605 10820 log_verifier.cc:177] Verified matching terms for 633 ops in tablet 5e130a7b478f4f91af11bef82ebf7236
I20240503 14:13:43.178656 10820 log_verifier.cc:126] Checking tablet 64d1257f67b54fb3a4b36f0f2b9f58c3
I20240503 14:13:43.598589 10820 log_verifier.cc:177] Verified matching terms for 438 ops in tablet 64d1257f67b54fb3a4b36f0f2b9f58c3
I20240503 14:13:43.599996 10820 log_verifier.cc:126] Checking tablet 71289cddae864085beacce5aa1fb0c4e
I20240503 14:13:44.193862 10820 log_verifier.cc:177] Verified matching terms for 623 ops in tablet 71289cddae864085beacce5aa1fb0c4e
I20240503 14:13:44.195345 10820 log_verifier.cc:126] Checking tablet 7778260935e54e7d90e6c8d929b2c564
I20240503 14:13:44.580566 10820 log_verifier.cc:177] Verified matching terms for 395 ops in tablet 7778260935e54e7d90e6c8d929b2c564
I20240503 14:13:44.581544 10820 log_verifier.cc:126] Checking tablet 99d215a7d95a41c6a14e7165cf0fe884
I20240503 14:13:44.983346 10820 log_verifier.cc:177] Verified matching terms for 377 ops in tablet 99d215a7d95a41c6a14e7165cf0fe884
I20240503 14:13:44.984295 10820 log_verifier.cc:126] Checking tablet a324330d54ff4a01982e13d0cc191296
I20240503 14:13:45.484889 10820 log_verifier.cc:177] Verified matching terms for 493 ops in tablet a324330d54ff4a01982e13d0cc191296
I20240503 14:13:45.486110 10820 log_verifier.cc:126] Checking tablet a959a6c207af45009ff7807ea6f20ee2
I20240503 14:13:46.147729 10820 log_verifier.cc:177] Verified matching terms for 734 ops in tablet a959a6c207af45009ff7807ea6f20ee2
I20240503 14:13:46.149765 10820 log_verifier.cc:126] Checking tablet c43a1cb5209c4a69a95877c842460adc
I20240503 14:13:46.549985 10820 log_verifier.cc:177] Verified matching terms for 442 ops in tablet c43a1cb5209c4a69a95877c842460adc
I20240503 14:13:46.551283 10820 log_verifier.cc:126] Checking tablet e82229abb5be4ed1abc3fa200671416e
I20240503 14:13:46.597256 10820 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e82229abb5be4ed1abc3fa200671416e
I20240503 14:13:46.597386 10820 log_verifier.cc:126] Checking tablet f99a260ae30a4efda1126bf96b1ac02f
I20240503 14:13:46.963397 10820 log_verifier.cc:177] Verified matching terms for 383 ops in tablet f99a260ae30a4efda1126bf96b1ac02f
I20240503 14:13:46.978912 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.8:0
--local_ip_for_outbound_sockets=127.10.145.8
--webserver_interface=127.10.145.8
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--builtin_ntp_servers=127.10.145.20:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:13:47.400663 16896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:47.401059 16896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:47.401177 16896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:47.409888 16896 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:47.410039 16896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:47.410192 16896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.8
I20240503 14:13:47.421079 16896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.10.145.8
--webserver_port=0
--tserver_master_addrs=127.10.145.62:35023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.8
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:47.422631 16896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:47.424839 16896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:47.439100 16903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:47.439594 16904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:47.439932 16906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:47.440392 16896 server_base.cc:1034] running on GCE node
I20240503 14:13:47.441078 16896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:47.442378 16896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:47.443624 16896 hybrid_clock.cc:648] HybridClock initialized: now 1714745627443549 us; error 70 us; skew 500 ppm
I20240503 14:13:47.443989 16896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:47.451404 16896 webserver.cc:460] Webserver started at http://127.10.145.8:37861/ using document root <none> and password file <none>
I20240503 14:13:47.452219 16896 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:47.452324 16896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:47.452632 16896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:47.455624 16896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "6a7786296dae40d4aa2c86869ddfb919"
format_stamp: "Formatted at 2024-05-03 14:13:47 on dist-test-slave-v549"
I20240503 14:13:47.456430 16896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "6a7786296dae40d4aa2c86869ddfb919"
format_stamp: "Formatted at 2024-05-03 14:13:47 on dist-test-slave-v549"
I20240503 14:13:47.462652 16896 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240503 14:13:47.467087 16912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:47.468675 16896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:47.468845 16896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6a7786296dae40d4aa2c86869ddfb919"
format_stamp: "Formatted at 2024-05-03 14:13:47 on dist-test-slave-v549"
I20240503 14:13:47.469044 16896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:47.490737 16896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:47.493072 16896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:47.493634 16896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:47.494874 16896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:47.496727 16896 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:47.496845 16896 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:47.496958 16896 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:47.497035 16896 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:47.536854 16896 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.8:34135
I20240503 14:13:47.536919 17024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.8:34135 every 8 connection(s)
I20240503 14:13:47.538782 16896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:13:47.546490 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 16896
I20240503 14:13:47.546685 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:13:47.553944 17025 heartbeater.cc:347] Connected to a master server at 127.10.145.62:35023
I20240503 14:13:47.554312 17025 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:47.555294 17025 heartbeater.cc:511] Master 127.10.145.62:35023 requested a full tablet report, sending...
I20240503 14:13:47.556957 12738 ts_manager.cc:194] Registered new tserver with Master: 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135)
I20240503 14:13:47.557860 12738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.8:43341
I20240503 14:13:48.157194 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:48.160730 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.161779 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:48.180691 16960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.195756 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:48.252799 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:48.254714 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:48.289525 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:48.512916 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.536713 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:48.557972 16960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:48.558375 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:48.561295 17025 heartbeater.cc:503] Master 127.10.145.62:35023 was elected leader, sending a full tablet report...
I20240503 14:13:48.594380 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:48.650269 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:48.656301 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:48.678512 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:48.771556 13399 consensus_queue.cc:237] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 395, Committed index: 395, Last appended: 1.395, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:48.775256 12877 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20240503 14:13:48.775815 13142 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20240503 14:13:48.775934 17122 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.000s
I20240503 14:13:48.776512 17123 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.000s
I20240503 14:13:48.777491 13273 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20240503 14:13:48.778455 14506 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20240503 14:13:48.778353 16893 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.000s
I20240503 14:13:48.779686 16893 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.000s
W20240503 14:13:48.786346 13338 consensus_peers.cc:485] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 -> Peer 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135): Couldn't send request to peer 6a7786296dae40d4aa2c86869ddfb919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7778260935e54e7d90e6c8d929b2c564. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:48.787160 16893 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.396: config changed from index 127 to 396, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.789113 13273 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 127 to 396, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.789490 12877 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 127 to 396, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.790544 13142 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 127 to 396, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.795441 14506 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 127 to 396, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.796455 12733 catalog_manager.cc:5484] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index 127 to 396, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.820441 13273 consensus_queue.cc:237] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 735, Committed index: 735, Last appended: 1.735, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:48.824467 13009 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 735. Preceding OpId from leader: term: 1 index: 736. (index mismatch)
I20240503 14:13:48.824519 13530 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 735. Preceding OpId from leader: term: 1 index: 736. (index mismatch)
I20240503 14:13:48.825129 16897 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 736, Last known committed idx: 735, Time since last communication: 0.000s
I20240503 14:13:48.826175 17112 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 736, Last known committed idx: 735, Time since last communication: 0.000s
I20240503 14:13:48.826844 14506 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 735. Preceding OpId from leader: term: 1 index: 736. (index mismatch)
I20240503 14:13:48.828079 17112 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 736, Last known committed idx: 735, Time since last communication: 0.000s
W20240503 14:13:48.832298 13208 consensus_peers.cc:485] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 -> Peer 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135): Couldn't send request to peer 6a7786296dae40d4aa2c86869ddfb919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32029fdaf54f4bccac238b11019ec913. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:48.834013 17111 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.736: config changed from index 358 to 736, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.835397 14506 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.736: config changed from index 358 to 736, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.838045 13142 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 735. Preceding OpId from leader: term: 1 index: 736. (index mismatch)
I20240503 14:13:48.838430 13530 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.736: config changed from index 358 to 736, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.840981 12737 catalog_manager.cc:5484] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de reported cstate change: config changed from index 358 to 736, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.841900 16897 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 736, Last known committed idx: 735, Time since last communication: 0.000s
I20240503 14:13:48.846812 13009 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.736: config changed from index 358 to 736, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.860538 13142 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.736: config changed from index 358 to 736, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.885641 12877 consensus_queue.cc:237] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 438, Committed index: 438, Last appended: 1.438, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:48.889648 13009 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 438. Preceding OpId from leader: term: 1 index: 439. (index mismatch)
I20240503 14:13:48.890288 16890 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 439, Last known committed idx: 438, Time since last communication: 0.000s
I20240503 14:13:48.898804 13530 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 438. Preceding OpId from leader: term: 1 index: 439. (index mismatch)
I20240503 14:13:48.900202 13399 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 438. Preceding OpId from leader: term: 1 index: 439. (index mismatch)
I20240503 14:13:48.900336 13273 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 438. Preceding OpId from leader: term: 1 index: 439. (index mismatch)
I20240503 14:13:48.900768 17153 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 439, Last known committed idx: 438, Time since last communication: 0.001s
I20240503 14:13:48.901415 17154 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 439, Last known committed idx: 438, Time since last communication: 0.000s
I20240503 14:13:48.901796 16890 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 439, Last known committed idx: 438, Time since last communication: 0.000s
W20240503 14:13:48.909379 12817 consensus_peers.cc:485] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 -> Peer 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135): Couldn't send request to peer 6a7786296dae40d4aa2c86869ddfb919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64d1257f67b54fb3a4b36f0f2b9f58c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:48.912226 16890 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 1 LEADER]: Committing config change with OpId 1.439: config changed from index -1 to 439, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.913539 13530 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.439: config changed from index -1 to 439, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.919867 12742 catalog_manager.cc:5484] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index -1 to 439, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "72b07e9b53b9471989185277045db8f7" committed_config { opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.921871 13009 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.439: config changed from index -1 to 439, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.922700 13399 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.439: config changed from index -1 to 439, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.934046 13273 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.439: config changed from index -1 to 439, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.949028 13530 consensus_queue.cc:237] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 623, Committed index: 623, Last appended: 1.623, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 624 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:48.952415 13009 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 623. Preceding OpId from leader: term: 1 index: 624. (index mismatch)
I20240503 14:13:48.953358 13142 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 623. Preceding OpId from leader: term: 1 index: 624. (index mismatch)
I20240503 14:13:48.953724 16815 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 624, Last known committed idx: 623, Time since last communication: 0.000s
I20240503 14:13:48.954059 17166 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 624, Last known committed idx: 623, Time since last communication: 0.000s
I20240503 14:13:48.954684 14506 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 623. Preceding OpId from leader: term: 1 index: 624. (index mismatch)
I20240503 14:13:48.954797 13399 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 623. Preceding OpId from leader: term: 1 index: 624. (index mismatch)
I20240503 14:13:48.956172 16815 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 624, Last known committed idx: 623, Time since last communication: 0.000s
I20240503 14:13:48.956688 16815 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 624, Last known committed idx: 623, Time since last communication: 0.000s
W20240503 14:13:48.962786 13468 consensus_peers.cc:485] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 -> Peer 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135): Couldn't send request to peer 6a7786296dae40d4aa2c86869ddfb919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71289cddae864085beacce5aa1fb0c4e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:48.979300 16815 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.624: config changed from index 437 to 624, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 624 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.986536 14506 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.624: config changed from index 437 to 624, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 624 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.993223 12742 catalog_manager.cc:5484] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de reported cstate change: config changed from index 437 to 624, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 624 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:48.995980 13009 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.624: config changed from index 437 to 624, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 624 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:49.006801 13399 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.624: config changed from index 437 to 624, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 624 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:49.010509 13142 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.624: config changed from index 437 to 624, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 624 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:49.029843 13273 consensus_queue.cc:237] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 442, Committed index: 442, Last appended: 1.442, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:49.032994 13399 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240503 14:13:49.032974 13142 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240503 14:13:49.033072 14506 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240503 14:13:49.033568 17111 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240503 14:13:49.033857 17112 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
W20240503 14:13:49.034121 13208 consensus_peers.cc:485] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 -> Peer 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135): Couldn't send request to peer 6a7786296dae40d4aa2c86869ddfb919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c43a1cb5209c4a69a95877c842460adc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:49.034193 16897 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240503 14:13:49.035425 12877 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 1 index: 443. (index mismatch)
I20240503 14:13:49.036067 17111 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240503 14:13:49.042495 16897 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.443: config changed from index 267 to 443, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:49.043476 13142 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 267 to 443, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:49.044168 14506 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 267 to 443, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:49.051671 12737 catalog_manager.cc:5484] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: config changed from index 267 to 443, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:49.053593 13399 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 267 to 443, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:49.060503 12877 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.443: config changed from index 267 to 443, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:49.294026 17210 ts_tablet_manager.cc:902] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: Initiating tablet copy from peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531)
I20240503 14:13:49.295951 17210 tablet_copy_client.cc:308] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:38531
I20240503 14:13:49.316192 12903 tablet_copy_service.cc:140] P 72b07e9b53b9471989185277045db8f7: Received BeginTabletCopySession request for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 from peer 6a7786296dae40d4aa2c86869ddfb919 ({username='slave'} at 127.10.145.8:56449)
I20240503 14:13:49.316411 12903 tablet_copy_service.cc:161] P 72b07e9b53b9471989185277045db8f7: Beginning new tablet copy session on tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 from peer 6a7786296dae40d4aa2c86869ddfb919 at {username='slave'} at 127.10.145.8:56449: session id = 6a7786296dae40d4aa2c86869ddfb919-64d1257f67b54fb3a4b36f0f2b9f58c3
I20240503 14:13:49.318862 12903 tablet_copy_source_session.cc:205] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.324620 17210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.339643 17210 tablet_copy_client.cc:783] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.340011 17210 tablet_copy_client.cc:647] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.345752 17210 tablet_copy_client.cc:523] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.351043 17210 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap starting.
I20240503 14:13:49.365928 17227 ts_tablet_manager.cc:902] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919: Initiating tablet copy from peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:13:49.370383 17230 ts_tablet_manager.cc:902] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919: Initiating tablet copy from peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:13:49.374023 17227 tablet_copy_client.cc:308] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:42597
I20240503 14:13:49.381826 13294 tablet_copy_service.cc:140] P 2cd2acf7d8274b5a95482dffd0823a08: Received BeginTabletCopySession request for tablet 32029fdaf54f4bccac238b11019ec913 from peer 6a7786296dae40d4aa2c86869ddfb919 ({username='slave'} at 127.10.145.8:54087)
I20240503 14:13:49.382043 13294 tablet_copy_service.cc:161] P 2cd2acf7d8274b5a95482dffd0823a08: Beginning new tablet copy session on tablet 32029fdaf54f4bccac238b11019ec913 from peer 6a7786296dae40d4aa2c86869ddfb919 at {username='slave'} at 127.10.145.8:54087: session id = 6a7786296dae40d4aa2c86869ddfb919-32029fdaf54f4bccac238b11019ec913
I20240503 14:13:49.384989 13294 tablet_copy_source_session.cc:205] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.387770 17227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32029fdaf54f4bccac238b11019ec913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.395608 17227 tablet_copy_client.cc:783] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.395929 17227 tablet_copy_client.cc:647] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.401746 17227 tablet_copy_client.cc:523] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.406028 17227 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap starting.
I20240503 14:13:49.421326 17230 tablet_copy_client.cc:308] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.5:38005
I20240503 14:13:49.429872 13424 tablet_copy_service.cc:140] P e148d8a3757340ab9288ea98cabb5ad3: Received BeginTabletCopySession request for tablet 7778260935e54e7d90e6c8d929b2c564 from peer 6a7786296dae40d4aa2c86869ddfb919 ({username='slave'} at 127.10.145.8:40893)
I20240503 14:13:49.430097 13424 tablet_copy_service.cc:161] P e148d8a3757340ab9288ea98cabb5ad3: Beginning new tablet copy session on tablet 7778260935e54e7d90e6c8d929b2c564 from peer 6a7786296dae40d4aa2c86869ddfb919 at {username='slave'} at 127.10.145.8:40893: session id = 6a7786296dae40d4aa2c86869ddfb919-7778260935e54e7d90e6c8d929b2c564
I20240503 14:13:49.433012 13424 tablet_copy_source_session.cc:205] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.436995 17230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7778260935e54e7d90e6c8d929b2c564. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.445101 17230 tablet_copy_client.cc:783] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.445518 17230 tablet_copy_client.cc:647] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.449952 17230 tablet_copy_client.cc:523] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.453585 17210 log.cc:824] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:49.453828 17230 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap starting.
I20240503 14:13:49.529567 17254 ts_tablet_manager.cc:902] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919: Initiating tablet copy from peer af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:13:49.588851 17260 ts_tablet_manager.cc:902] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919: Initiating tablet copy from peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:13:49.591454 17254 tablet_copy_client.cc:308] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36377
I20240503 14:13:49.602535 13554 tablet_copy_service.cc:140] P af88d71da3a54f779cf4f651f4393bb6: Received BeginTabletCopySession request for tablet 71289cddae864085beacce5aa1fb0c4e from peer 6a7786296dae40d4aa2c86869ddfb919 ({username='slave'} at 127.10.145.8:35377)
I20240503 14:13:49.602766 13554 tablet_copy_service.cc:161] P af88d71da3a54f779cf4f651f4393bb6: Beginning new tablet copy session on tablet 71289cddae864085beacce5aa1fb0c4e from peer 6a7786296dae40d4aa2c86869ddfb919 at {username='slave'} at 127.10.145.8:35377: session id = 6a7786296dae40d4aa2c86869ddfb919-71289cddae864085beacce5aa1fb0c4e
I20240503 14:13:49.608050 13554 tablet_copy_source_session.cc:205] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.609966 17254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71289cddae864085beacce5aa1fb0c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.618142 17254 tablet_copy_client.cc:783] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.618418 17254 tablet_copy_client.cc:647] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.622900 17254 tablet_copy_client.cc:523] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.626783 17254 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap starting.
I20240503 14:13:49.639127 17260 tablet_copy_client.cc:308] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:42597
I20240503 14:13:49.641712 13294 tablet_copy_service.cc:140] P 2cd2acf7d8274b5a95482dffd0823a08: Received BeginTabletCopySession request for tablet c43a1cb5209c4a69a95877c842460adc from peer 6a7786296dae40d4aa2c86869ddfb919 ({username='slave'} at 127.10.145.8:54087)
I20240503 14:13:49.641887 13294 tablet_copy_service.cc:161] P 2cd2acf7d8274b5a95482dffd0823a08: Beginning new tablet copy session on tablet c43a1cb5209c4a69a95877c842460adc from peer 6a7786296dae40d4aa2c86869ddfb919 at {username='slave'} at 127.10.145.8:54087: session id = 6a7786296dae40d4aa2c86869ddfb919-c43a1cb5209c4a69a95877c842460adc
I20240503 14:13:49.644711 13294 tablet_copy_source_session.cc:205] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.646340 17260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43a1cb5209c4a69a95877c842460adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.653666 17260 tablet_copy_client.cc:783] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.654053 17260 tablet_copy_client.cc:647] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.659092 17260 tablet_copy_client.cc:523] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.662699 17260 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap starting.
I20240503 14:13:50.389840 17210 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap replayed 1/1 log segments. Stats: ops{read=439 overwritten=0 applied=439 ignored=0} inserts{seen=1111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:50.390583 17210 tablet_bootstrap.cc:492] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap complete.
I20240503 14:13:50.391214 17210 ts_tablet_manager.cc:1367] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: Time spent bootstrapping tablet: real 1.040s user 0.559s sys 0.024s
I20240503 14:13:50.411960 17210 raft_consensus.cc:348] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:50.412747 17210 raft_consensus.cc:729] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a7786296dae40d4aa2c86869ddfb919, State: Initialized, Role: LEARNER
I20240503 14:13:50.413539 17210 consensus_queue.cc:260] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 439, Last appended: 1.439, Last appended by leader: 439, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:50.438741 17210 ts_tablet_manager.cc:1398] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919: Time spent starting tablet: real 0.047s user 0.020s sys 0.008s
I20240503 14:13:50.441665 12903 tablet_copy_service.cc:342] P 72b07e9b53b9471989185277045db8f7: Request end of tablet copy session 6a7786296dae40d4aa2c86869ddfb919-64d1257f67b54fb3a4b36f0f2b9f58c3 received from {username='slave'} at 127.10.145.8:56449
I20240503 14:13:50.441867 12903 tablet_copy_service.cc:433] P 72b07e9b53b9471989185277045db8f7: ending tablet copy session 6a7786296dae40d4aa2c86869ddfb919-64d1257f67b54fb3a4b36f0f2b9f58c3 on tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 with peer 6a7786296dae40d4aa2c86869ddfb919
I20240503 14:13:50.473026 17230 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap replayed 1/1 log segments. Stats: ops{read=396 overwritten=0 applied=396 ignored=0} inserts{seen=1065 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:50.473842 17230 tablet_bootstrap.cc:492] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap complete.
I20240503 14:13:50.474385 17230 ts_tablet_manager.cc:1367] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919: Time spent bootstrapping tablet: real 1.021s user 0.472s sys 0.045s
I20240503 14:13:50.475634 17230 raft_consensus.cc:348] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:50.476133 17230 raft_consensus.cc:729] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a7786296dae40d4aa2c86869ddfb919, State: Initialized, Role: LEARNER
I20240503 14:13:50.476619 17230 consensus_queue.cc:260] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 396, Last appended: 1.396, Last appended by leader: 396, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:50.477429 17230 ts_tablet_manager.cc:1398] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:50.478368 13424 tablet_copy_service.cc:342] P e148d8a3757340ab9288ea98cabb5ad3: Request end of tablet copy session 6a7786296dae40d4aa2c86869ddfb919-7778260935e54e7d90e6c8d929b2c564 received from {username='slave'} at 127.10.145.8:40893
I20240503 14:13:50.478489 13424 tablet_copy_service.cc:433] P e148d8a3757340ab9288ea98cabb5ad3: ending tablet copy session 6a7786296dae40d4aa2c86869ddfb919-7778260935e54e7d90e6c8d929b2c564 on tablet 7778260935e54e7d90e6c8d929b2c564 with peer 6a7786296dae40d4aa2c86869ddfb919
I20240503 14:13:50.499027 16980 raft_consensus.cc:1203] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.438->[1.439-1.439] Dedup: 1.439->[]
I20240503 14:13:50.604427 16980 raft_consensus.cc:1203] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.395->[1.396-1.396] Dedup: 1.396->[]
I20240503 14:13:50.660884 17260 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap replayed 1/1 log segments. Stats: ops{read=443 overwritten=0 applied=443 ignored=0} inserts{seen=1125 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:50.661491 17260 tablet_bootstrap.cc:492] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap complete.
I20240503 14:13:50.661904 17260 ts_tablet_manager.cc:1367] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919: Time spent bootstrapping tablet: real 0.999s user 0.493s sys 0.062s
I20240503 14:13:50.662885 17260 raft_consensus.cc:348] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:50.663273 17260 raft_consensus.cc:729] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a7786296dae40d4aa2c86869ddfb919, State: Initialized, Role: LEARNER
I20240503 14:13:50.663585 17260 consensus_queue.cc:260] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 443, Last appended: 1.443, Last appended by leader: 443, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:50.664412 17260 ts_tablet_manager.cc:1398] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:50.665221 13294 tablet_copy_service.cc:342] P 2cd2acf7d8274b5a95482dffd0823a08: Request end of tablet copy session 6a7786296dae40d4aa2c86869ddfb919-c43a1cb5209c4a69a95877c842460adc received from {username='slave'} at 127.10.145.8:54087
I20240503 14:13:50.665375 13294 tablet_copy_service.cc:433] P 2cd2acf7d8274b5a95482dffd0823a08: ending tablet copy session 6a7786296dae40d4aa2c86869ddfb919-c43a1cb5209c4a69a95877c842460adc on tablet c43a1cb5209c4a69a95877c842460adc with peer 6a7786296dae40d4aa2c86869ddfb919
I20240503 14:13:50.765636 17227 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap replayed 1/1 log segments. Stats: ops{read=736 overwritten=0 applied=736 ignored=0} inserts{seen=1418 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:50.766067 17227 tablet_bootstrap.cc:492] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap complete.
I20240503 14:13:50.766403 17227 ts_tablet_manager.cc:1367] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919: Time spent bootstrapping tablet: real 1.361s user 0.774s sys 0.104s
I20240503 14:13:50.767205 17227 raft_consensus.cc:348] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:50.767616 17227 raft_consensus.cc:729] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a7786296dae40d4aa2c86869ddfb919, State: Initialized, Role: LEARNER
I20240503 14:13:50.767881 17227 consensus_queue.cc:260] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 736, Last appended: 1.736, Last appended by leader: 736, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:50.769428 17227 ts_tablet_manager.cc:1398] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:50.770437 13294 tablet_copy_service.cc:342] P 2cd2acf7d8274b5a95482dffd0823a08: Request end of tablet copy session 6a7786296dae40d4aa2c86869ddfb919-32029fdaf54f4bccac238b11019ec913 received from {username='slave'} at 127.10.145.8:54087
I20240503 14:13:50.770560 13294 tablet_copy_service.cc:433] P 2cd2acf7d8274b5a95482dffd0823a08: ending tablet copy session 6a7786296dae40d4aa2c86869ddfb919-32029fdaf54f4bccac238b11019ec913 on tablet 32029fdaf54f4bccac238b11019ec913 with peer 6a7786296dae40d4aa2c86869ddfb919
I20240503 14:13:50.821745 17254 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap replayed 1/1 log segments. Stats: ops{read=624 overwritten=0 applied=624 ignored=0} inserts{seen=1249 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:50.822247 17254 tablet_bootstrap.cc:492] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap complete.
I20240503 14:13:50.822585 17254 ts_tablet_manager.cc:1367] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919: Time spent bootstrapping tablet: real 1.196s user 0.659s sys 0.095s
I20240503 14:13:50.823233 17254 raft_consensus.cc:348] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 624 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:50.823479 17254 raft_consensus.cc:729] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a7786296dae40d4aa2c86869ddfb919, State: Initialized, Role: LEARNER
I20240503 14:13:50.823673 17254 consensus_queue.cc:260] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 624, Last appended: 1.624, Last appended by leader: 624, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 624 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:50.824138 17254 ts_tablet_manager.cc:1398] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:50.824949 13554 tablet_copy_service.cc:342] P af88d71da3a54f779cf4f651f4393bb6: Request end of tablet copy session 6a7786296dae40d4aa2c86869ddfb919-71289cddae864085beacce5aa1fb0c4e received from {username='slave'} at 127.10.145.8:35377
I20240503 14:13:50.825068 13554 tablet_copy_service.cc:433] P af88d71da3a54f779cf4f651f4393bb6: ending tablet copy session 6a7786296dae40d4aa2c86869ddfb919-71289cddae864085beacce5aa1fb0c4e on tablet 71289cddae864085beacce5aa1fb0c4e with peer 6a7786296dae40d4aa2c86869ddfb919
I20240503 14:13:50.900226 17288 raft_consensus.cc:1050] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7: attempting to promote NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 to VOTER
I20240503 14:13:50.901393 17288 consensus_queue.cc:237] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 439, Committed index: 439, Last appended: 1.439, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:50.904232 13009 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 439. Preceding OpId from leader: term: 1 index: 440. (index mismatch)
I20240503 14:13:50.904412 16980 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 439. Preceding OpId from leader: term: 1 index: 440. (index mismatch)
I20240503 14:13:50.904844 13273 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 439. Preceding OpId from leader: term: 1 index: 440. (index mismatch)
I20240503 14:13:50.905129 17358 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 440, Last known committed idx: 439, Time since last communication: 0.000s
I20240503 14:13:50.905613 17285 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 440, Last known committed idx: 439, Time since last communication: 0.000s
I20240503 14:13:50.905925 17360 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 440, Last known committed idx: 439, Time since last communication: 0.000s
I20240503 14:13:50.908325 13530 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 439. Preceding OpId from leader: term: 1 index: 440. (index mismatch)
I20240503 14:13:50.914330 17358 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 440, Last known committed idx: 439, Time since last communication: 0.000s
I20240503 14:13:50.916451 13398 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 439. Preceding OpId from leader: term: 1 index: 440. (index mismatch)
I20240503 14:13:50.917273 17361 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 440, Last known committed idx: 439, Time since last communication: 0.000s
I20240503 14:13:50.921025 17361 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 1 LEADER]: Committing config change with OpId 1.440: config changed from index 439 to 440, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.921928 13009 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.440: config changed from index 439 to 440, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.922672 13398 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.440: config changed from index 439 to 440, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.923861 16980 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.440: config changed from index 439 to 440, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.926626 13273 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.440: config changed from index 439 to 440, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.926573 13530 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.440: config changed from index 439 to 440, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.930716 12737 catalog_manager.cc:5484] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 reported cstate change: config changed from index 439 to 440, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "72b07e9b53b9471989185277045db8f7" committed_config { opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:50.942101 12877 consensus_queue.cc:237] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 440, Committed index: 440, Last appended: 1.440, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:50.944764 16980 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 440. Preceding OpId from leader: term: 1 index: 441. (index mismatch)
I20240503 14:13:50.944743 13273 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 440. Preceding OpId from leader: term: 1 index: 441. (index mismatch)
I20240503 14:13:50.945343 17361 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I20240503 14:13:50.945863 17358 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I20240503 14:13:50.947328 13530 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 440. Preceding OpId from leader: term: 1 index: 441. (index mismatch)
I20240503 14:13:50.948176 17361 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I20240503 14:13:50.948069 13398 raft_consensus.cc:1261] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 72b07e9b53b9471989185277045db8f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 440. Preceding OpId from leader: term: 1 index: 441. (index mismatch)
I20240503 14:13:50.949229 17288 consensus_queue.cc:1035] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I20240503 14:13:50.949865 17288 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 72b07e9b53b9471989185277045db8f7 [term 1 LEADER]: Committing config change with OpId 1.441: config changed from index 440 to 441, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.950543 13273 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.441: config changed from index 440 to 441, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.951504 13530 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.441: config changed from index 440 to 441, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.953159 13399 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.441: config changed from index 440 to 441, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.953698 12726 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 with cas_config_opid_index 440: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:50.956486 12735 catalog_manager.cc:5484] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index 440 to 441, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "72b07e9b53b9471989185277045db8f7" committed_config { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.958400 16980 raft_consensus.cc:2932] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.441: config changed from index 440 to 441, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:50.961582 12992 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 with delete_type TABLET_DATA_TOMBSTONED (TS 3678714622584fa4883e4bf07fcabd75 not found in new config with opid_index 441) from {username='slave'} at 127.0.0.1:55928
I20240503 14:13:50.963268 17371 tablet_replica.cc:330] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75: stopping tablet replica
I20240503 14:13:50.967041 17371 raft_consensus.cc:2229] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:50.967784 16980 raft_consensus.cc:1203] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.623->[1.624-1.624] Dedup: 1.624->[]
I20240503 14:13:50.967926 17371 raft_consensus.cc:2258] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:50.970798 17371 ts_tablet_manager.cc:1875] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:50.975703 17371 ts_tablet_manager.cc:1888] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.440
I20240503 14:13:50.975845 17371 log.cc:1196] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 3678714622584fa4883e4bf07fcabd75: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/64d1257f67b54fb3a4b36f0f2b9f58c3
I20240503 14:13:50.976684 12726 catalog_manager.cc:4830] TS 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): tablet 64d1257f67b54fb3a4b36f0f2b9f58c3 (table rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]) successfully deleted
I20240503 14:13:51.056561 16980 raft_consensus.cc:1203] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.442->[1.443-1.443] Dedup: 1.443->[]
I20240503 14:13:51.062021 13530 consensus_queue.cc:237] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383, Committed index: 383, Last appended: 1.383, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:51.064503 16980 raft_consensus.cc:1203] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.735->[1.736-1.736] Dedup: 1.736->[]
W20240503 14:13:51.065563 13468 consensus_peers.cc:485] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 -> Peer 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135): Couldn't send request to peer 6a7786296dae40d4aa2c86869ddfb919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f99a260ae30a4efda1126bf96b1ac02f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:51.065804 12877 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 383. Preceding OpId from leader: term: 1 index: 384. (index mismatch)
I20240503 14:13:51.066241 13398 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 383. Preceding OpId from leader: term: 1 index: 384. (index mismatch)
I20240503 14:13:51.066354 13009 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 383. Preceding OpId from leader: term: 1 index: 384. (index mismatch)
I20240503 14:13:51.066978 17302 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20240503 14:13:51.067418 17403 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20240503 14:13:51.068079 17304 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20240503 14:13:51.071436 17302 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.384: config changed from index 135 to 384, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.072060 12877 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.384: config changed from index 135 to 384, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.073273 13273 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 383. Preceding OpId from leader: term: 1 index: 384. (index mismatch)
I20240503 14:13:51.073137 13398 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.384: config changed from index 135 to 384, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.073956 16815 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20240503 14:13:51.078390 12733 catalog_manager.cc:5484] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 reported cstate change: config changed from index 135 to 384, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.086120 13009 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.384: config changed from index 135 to 384, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.094094 13273 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.384: config changed from index 135 to 384, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.129061 17419 raft_consensus.cc:1050] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3: attempting to promote NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 to VOTER
I20240503 14:13:51.130122 17419 consensus_queue.cc:237] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 396, Committed index: 396, Last appended: 1.396, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:51.133795 13273 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 396. Preceding OpId from leader: term: 1 index: 397. (index mismatch)
I20240503 14:13:51.133827 16979 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 396. Preceding OpId from leader: term: 1 index: 397. (index mismatch)
I20240503 14:13:51.134335 13142 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 396. Preceding OpId from leader: term: 1 index: 397. (index mismatch)
I20240503 14:13:51.134495 12877 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 396. Preceding OpId from leader: term: 1 index: 397. (index mismatch)
I20240503 14:13:51.134650 17413 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I20240503 14:13:51.135195 17129 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I20240503 14:13:51.136345 14506 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 396. Preceding OpId from leader: term: 1 index: 397. (index mismatch)
I20240503 14:13:51.137073 17129 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I20240503 14:13:51.137610 17129 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I20240503 14:13:51.139612 17129 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I20240503 14:13:51.155827 17129 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.397: config changed from index 396 to 397, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.156916 14506 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.397: config changed from index 396 to 397, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.161316 13273 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.397: config changed from index 396 to 397, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.161486 12877 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.397: config changed from index 396 to 397, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.165722 16980 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.397: config changed from index 396 to 397, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.167410 13142 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.397: config changed from index 396 to 397, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.170308 12739 catalog_manager.cc:5484] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 reported cstate change: config changed from index 396 to 397, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:51.183938 13398 consensus_queue.cc:237] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 397, Committed index: 397, Last appended: 1.397, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:51.186821 12877 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 397. Preceding OpId from leader: term: 1 index: 398. (index mismatch)
I20240503 14:13:51.187458 13142 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 397. Preceding OpId from leader: term: 1 index: 398. (index mismatch)
I20240503 14:13:51.188187 17419 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 398, Last known committed idx: 397, Time since last communication: 0.000s
I20240503 14:13:51.188978 16980 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 397. Preceding OpId from leader: term: 1 index: 398. (index mismatch)
I20240503 14:13:51.189821 17421 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 398, Last known committed idx: 397, Time since last communication: 0.000s
I20240503 14:13:51.190507 14506 raft_consensus.cc:1261] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 397. Preceding OpId from leader: term: 1 index: 398. (index mismatch)
I20240503 14:13:51.190459 17421 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 398, Last known committed idx: 397, Time since last communication: 0.000s
I20240503 14:13:51.191745 17419 consensus_queue.cc:1035] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 398, Last known committed idx: 397, Time since last communication: 0.000s
I20240503 14:13:51.193625 17129 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.398: config changed from index 397 to 398, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.194379 12877 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.398: config changed from index 397 to 398, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.195619 16979 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.398: config changed from index 397 to 398, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.200194 12738 catalog_manager.cc:5484] T 7778260935e54e7d90e6c8d929b2c564 P 72b07e9b53b9471989185277045db8f7 reported cstate change: config changed from index 397 to 398, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.201808 12727 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7778260935e54e7d90e6c8d929b2c564 with cas_config_opid_index 397: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:51.202587 14506 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.398: config changed from index 397 to 398, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.205214 13254 tablet_service.cc:1514] Processing DeleteTablet for tablet 7778260935e54e7d90e6c8d929b2c564 with delete_type TABLET_DATA_TOMBSTONED (TS 2cd2acf7d8274b5a95482dffd0823a08 not found in new config with opid_index 398) from {username='slave'} at 127.0.0.1:56270
I20240503 14:13:51.210628 13142 raft_consensus.cc:2932] T 7778260935e54e7d90e6c8d929b2c564 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.398: config changed from index 397 to 398, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.213914 17441 tablet_replica.cc:330] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08: stopping tablet replica
I20240503 14:13:51.214116 17441 raft_consensus.cc:2229] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:51.214326 17441 raft_consensus.cc:2258] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:51.218609 17441 ts_tablet_manager.cc:1875] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:51.223732 17441 ts_tablet_manager.cc:1888] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.397
I20240503 14:13:51.223872 17441 log.cc:1196] T 7778260935e54e7d90e6c8d929b2c564 P 2cd2acf7d8274b5a95482dffd0823a08: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/7778260935e54e7d90e6c8d929b2c564
I20240503 14:13:51.224725 12727 catalog_manager.cc:4830] TS 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): tablet 7778260935e54e7d90e6c8d929b2c564 (table rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]) successfully deleted
I20240503 14:13:51.448978 16815 raft_consensus.cc:1050] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6: attempting to promote NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 to VOTER
I20240503 14:13:51.449936 16815 consensus_queue.cc:237] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 624, Committed index: 624, Last appended: 1.624, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:51.451337 17299 raft_consensus.cc:1050] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08: attempting to promote NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 to VOTER
I20240503 14:13:51.452405 17299 consensus_queue.cc:237] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 443, Committed index: 443, Last appended: 1.443, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:51.455758 16979 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240503 14:13:51.457378 17462 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240503 14:13:51.459882 12878 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240503 14:13:51.460057 13398 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240503 14:13:51.460825 14506 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240503 14:13:51.461483 13142 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 443. Preceding OpId from leader: term: 1 index: 444. (index mismatch)
I20240503 14:13:51.462041 17291 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240503 14:13:51.462558 17465 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240503 14:13:51.463274 17465 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240503 14:13:51.465521 17291 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 443, Time since last communication: 0.000s
I20240503 14:13:51.471230 16979 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 624. Preceding OpId from leader: term: 1 index: 625. (index mismatch)
I20240503 14:13:51.471700 13398 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 624. Preceding OpId from leader: term: 1 index: 625. (index mismatch)
I20240503 14:13:51.472069 14506 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 624. Preceding OpId from leader: term: 1 index: 625. (index mismatch)
I20240503 14:13:51.472162 13009 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 624. Preceding OpId from leader: term: 1 index: 625. (index mismatch)
I20240503 14:13:51.473392 17302 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 625, Last known committed idx: 624, Time since last communication: 0.000s
I20240503 14:13:51.474364 17302 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 625, Last known committed idx: 624, Time since last communication: 0.000s
I20240503 14:13:51.474889 17304 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 625, Last known committed idx: 624, Time since last communication: 0.000s
I20240503 14:13:51.476083 13142 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 624. Preceding OpId from leader: term: 1 index: 625. (index mismatch)
I20240503 14:13:51.477500 16815 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 625, Last known committed idx: 624, Time since last communication: 0.000s
I20240503 14:13:51.480805 17302 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 625, Last known committed idx: 624, Time since last communication: 0.000s
I20240503 14:13:51.500340 17403 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.625: config changed from index 624 to 625, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.501721 17291 raft_consensus.cc:1050] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08: attempting to promote NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 to VOTER
I20240503 14:13:51.502849 17291 consensus_queue.cc:237] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 736, Committed index: 736, Last appended: 1.736, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:51.506767 17462 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.444: config changed from index 443 to 444, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.507629 13014 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 736. Preceding OpId from leader: term: 1 index: 737. (index mismatch)
I20240503 14:13:51.508396 17465 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 737, Last known committed idx: 736, Time since last communication: 0.000s
I20240503 14:13:51.509418 13142 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 736. Preceding OpId from leader: term: 1 index: 737. (index mismatch)
I20240503 14:13:51.508875 13009 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.625: config changed from index 624 to 625, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.509882 13399 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.625: config changed from index 624 to 625, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.511454 13530 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 736. Preceding OpId from leader: term: 1 index: 737. (index mismatch)
I20240503 14:13:51.512548 12733 catalog_manager.cc:5484] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 624 to 625, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:51.516788 17465 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 737, Last known committed idx: 736, Time since last communication: 0.000s
I20240503 14:13:51.517273 13140 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.625: config changed from index 624 to 625, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.518551 17465 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 737, Last known committed idx: 736, Time since last communication: 0.000s
I20240503 14:13:51.525050 13138 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.526330 16980 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 736. Preceding OpId from leader: term: 1 index: 737. (index mismatch)
I20240503 14:13:51.527195 13399 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.529930 14502 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 736. Preceding OpId from leader: term: 1 index: 737. (index mismatch)
I20240503 14:13:51.530439 17462 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 737, Last known committed idx: 736, Time since last communication: 0.000s
I20240503 14:13:51.530386 12878 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.537019 12738 catalog_manager.cc:5484] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index 443 to 444, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:51.540761 17462 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 737, Last known committed idx: 736, Time since last communication: 0.000s
I20240503 14:13:51.567488 16976 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.625: config changed from index 624 to 625, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.572921 14502 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.581570 14502 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.625: config changed from index 624 to 625, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.581501 16978 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.444: config changed from index 443 to 444, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.583025 13532 consensus_queue.cc:237] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 625, Committed index: 625, Last appended: 1.625, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 626 OBSOLETE_local: false peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:51.585036 13273 consensus_queue.cc:237] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 444, Committed index: 444, Last appended: 1.444, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:51.585778 13009 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 625. Preceding OpId from leader: term: 1 index: 626. (index mismatch)
I20240503 14:13:51.586242 13399 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 625. Preceding OpId from leader: term: 1 index: 626. (index mismatch)
I20240503 14:13:51.586899 16815 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 626, Last known committed idx: 625, Time since last communication: 0.000s
I20240503 14:13:51.588184 14503 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240503 14:13:51.588201 13138 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240503 14:13:51.588253 13399 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240503 14:13:51.588938 17462 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240503 14:13:51.589128 16976 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 625. Preceding OpId from leader: term: 1 index: 626. (index mismatch)
I20240503 14:13:51.589680 17291 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240503 14:13:51.589737 17302 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 626, Last known committed idx: 625, Time since last communication: 0.000s
I20240503 14:13:51.590257 17302 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 626, Last known committed idx: 625, Time since last communication: 0.000s
I20240503 14:13:51.592813 16980 raft_consensus.cc:1261] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch)
I20240503 14:13:51.596436 17291 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240503 14:13:51.596391 16815 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.626: config changed from index 625 to 626, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 626 OBSOLETE_local: false peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.596972 17462 consensus_queue.cc:1035] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I20240503 14:13:51.599681 17462 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.600067 13009 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.626: config changed from index 625 to 626, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 626 OBSOLETE_local: false peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.600601 14506 raft_consensus.cc:1261] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 625. Preceding OpId from leader: term: 1 index: 626. (index mismatch)
I20240503 14:13:51.601523 17304 consensus_queue.cc:1035] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 626, Last known committed idx: 625, Time since last communication: 0.000s
I20240503 14:13:51.603209 14506 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.626: config changed from index 625 to 626, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 626 OBSOLETE_local: false peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.606561 12727 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c43a1cb5209c4a69a95877c842460adc with cas_config_opid_index 444: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:51.606556 13399 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.607373 13140 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.610072 12733 catalog_manager.cc:5484] T 71289cddae864085beacce5aa1fb0c4e P 3678714622584fa4883e4bf07fcabd75 reported cstate change: config changed from index 625 to 626, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 626 OBSOLETE_local: false peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.614147 13398 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.626: config changed from index 625 to 626, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 626 OBSOLETE_local: false peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.614387 12735 catalog_manager.cc:5484] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index 444 to 445, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:51.615018 12729 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71289cddae864085beacce5aa1fb0c4e with cas_config_opid_index 625: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:51.612913 16978 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.619529 12863 tablet_service.cc:1514] Processing DeleteTablet for tablet c43a1cb5209c4a69a95877c842460adc with delete_type TABLET_DATA_TOMBSTONED (TS 72b07e9b53b9471989185277045db8f7 not found in new config with opid_index 445) from {username='slave'} at 127.0.0.1:43658
I20240503 14:13:51.625269 14506 raft_consensus.cc:2932] T c43a1cb5209c4a69a95877c842460adc P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.445: config changed from index 444 to 445, VOTER 72b07e9b53b9471989185277045db8f7 (127.10.145.1) evicted. New config: { opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.624953 16980 raft_consensus.cc:2932] T 71289cddae864085beacce5aa1fb0c4e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.626: config changed from index 625 to 626, VOTER 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3) evicted. New config: { opid_index: 626 OBSOLETE_local: false peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.635218 17486 tablet_replica.cc:330] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7: stopping tablet replica
I20240503 14:13:51.635547 17486 raft_consensus.cc:2229] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:51.635813 17486 raft_consensus.cc:2258] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:51.637207 17486 ts_tablet_manager.cc:1875] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:51.638855 13122 tablet_service.cc:1514] Processing DeleteTablet for tablet 71289cddae864085beacce5aa1fb0c4e with delete_type TABLET_DATA_TOMBSTONED (TS 7b75d8c9e35a416ba24798fde8b9def0 not found in new config with opid_index 626) from {username='slave'} at 127.0.0.1:34920
I20240503 14:13:51.644037 17488 tablet_replica.cc:330] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0: stopping tablet replica
I20240503 14:13:51.644263 17488 raft_consensus.cc:2229] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:51.646257 17485 ts_tablet_manager.cc:902] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919: Initiating tablet copy from peer af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
I20240503 14:13:51.645515 17462 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.737: config changed from index 736 to 737, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.647138 17488 raft_consensus.cc:2258] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:51.648136 13009 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.737: config changed from index 736 to 737, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.646091 17486 ts_tablet_manager.cc:1888] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.444
I20240503 14:13:51.649083 17486 log.cc:1196] T c43a1cb5209c4a69a95877c842460adc P 72b07e9b53b9471989185277045db8f7: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/c43a1cb5209c4a69a95877c842460adc
I20240503 14:13:51.650326 12726 catalog_manager.cc:4830] TS 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): tablet c43a1cb5209c4a69a95877c842460adc (table rebalance_test_table_0 [id=9a6d684b9b7f47f386c102fdfe70e8c4]) successfully deleted
I20240503 14:13:51.651175 17488 ts_tablet_manager.cc:1875] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:51.652755 14502 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.737: config changed from index 736 to 737, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.654508 12742 catalog_manager.cc:5484] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index 736 to 737, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:51.658519 13140 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.737: config changed from index 736 to 737, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.663794 13273 consensus_queue.cc:237] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 737, Committed index: 737, Last appended: 1.737, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:51.664391 17488 ts_tablet_manager.cc:1888] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.625
I20240503 14:13:51.664525 17488 log.cc:1196] T 71289cddae864085beacce5aa1fb0c4e P 7b75d8c9e35a416ba24798fde8b9def0: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/71289cddae864085beacce5aa1fb0c4e
I20240503 14:13:51.666153 12728 catalog_manager.cc:4830] TS 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): tablet 71289cddae864085beacce5aa1fb0c4e (table rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]) successfully deleted
I20240503 14:13:51.666760 14506 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 737. Preceding OpId from leader: term: 1 index: 738. (index mismatch)
I20240503 14:13:51.667012 16978 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 737. Preceding OpId from leader: term: 1 index: 738. (index mismatch)
I20240503 14:13:51.667430 17291 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 738, Last known committed idx: 737, Time since last communication: 0.000s
I20240503 14:13:51.667717 17463 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 738, Last known committed idx: 736, Time since last communication: 0.000s
I20240503 14:13:51.668396 13014 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 737. Preceding OpId from leader: term: 1 index: 738. (index mismatch)
I20240503 14:13:51.668864 16978 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.737: config changed from index 736 to 737, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.669440 17291 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 738, Last known committed idx: 737, Time since last communication: 0.000s
I20240503 14:13:51.672124 13138 raft_consensus.cc:1261] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Refusing update from remote peer 2cd2acf7d8274b5a95482dffd0823a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 737. Preceding OpId from leader: term: 1 index: 738. (index mismatch)
I20240503 14:13:51.671799 17291 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 LEADER]: Committing config change with OpId 1.738: config changed from index 737 to 738, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.672969 17463 consensus_queue.cc:1035] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 738, Last known committed idx: 737, Time since last communication: 0.000s
I20240503 14:13:51.674499 13014 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.738: config changed from index 737 to 738, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.674517 13138 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Committing config change with OpId 1.738: config changed from index 737 to 738, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.680073 12727 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32029fdaf54f4bccac238b11019ec913 with cas_config_opid_index 737: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:51.680799 14502 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.738: config changed from index 737 to 738, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.675148 16978 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.738: config changed from index 737 to 738, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.682091 12742 catalog_manager.cc:5484] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index 737 to 738, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "2cd2acf7d8274b5a95482dffd0823a08" committed_config { opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:51.646368 13530 raft_consensus.cc:2932] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.737: config changed from index 736 to 737, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:51.699666 13514 tablet_service.cc:1514] Processing DeleteTablet for tablet 32029fdaf54f4bccac238b11019ec913 with delete_type TABLET_DATA_TOMBSTONED (TS af88d71da3a54f779cf4f651f4393bb6 not found in new config with opid_index 738) from {username='slave'} at 127.0.0.1:48902
I20240503 14:13:51.703522 17493 tablet_replica.cc:330] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6: stopping tablet replica
I20240503 14:13:51.703800 17493 raft_consensus.cc:2229] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:51.704090 17493 raft_consensus.cc:2258] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:51.705518 17493 ts_tablet_manager.cc:1875] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:51.723958 17493 ts_tablet_manager.cc:1888] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.737
I20240503 14:13:51.724112 17493 log.cc:1196] T 32029fdaf54f4bccac238b11019ec913 P af88d71da3a54f779cf4f651f4393bb6: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/32029fdaf54f4bccac238b11019ec913
I20240503 14:13:51.728376 12729 catalog_manager.cc:4830] TS af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377): tablet 32029fdaf54f4bccac238b11019ec913 (table rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]) successfully deleted
I20240503 14:13:51.741616 17485 tablet_copy_client.cc:308] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36377
I20240503 14:13:51.742657 13554 tablet_copy_service.cc:140] P af88d71da3a54f779cf4f651f4393bb6: Received BeginTabletCopySession request for tablet f99a260ae30a4efda1126bf96b1ac02f from peer 6a7786296dae40d4aa2c86869ddfb919 ({username='slave'} at 127.10.145.8:35377)
I20240503 14:13:51.742827 13554 tablet_copy_service.cc:161] P af88d71da3a54f779cf4f651f4393bb6: Beginning new tablet copy session on tablet f99a260ae30a4efda1126bf96b1ac02f from peer 6a7786296dae40d4aa2c86869ddfb919 at {username='slave'} at 127.10.145.8:35377: session id = 6a7786296dae40d4aa2c86869ddfb919-f99a260ae30a4efda1126bf96b1ac02f
I20240503 14:13:51.745051 13554 tablet_copy_source_session.cc:205] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:51.746686 17485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99a260ae30a4efda1126bf96b1ac02f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.755872 17485 tablet_copy_client.cc:783] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:51.756179 17485 tablet_copy_client.cc:647] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:51.770026 17485 tablet_copy_client.cc:523] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:51.775530 17485 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap starting.
I20240503 14:13:51.792321 13142 consensus_queue.cc:237] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 633, Committed index: 633, Last appended: 1.633, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:51.795490 14502 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 633. Preceding OpId from leader: term: 1 index: 634. (index mismatch)
I20240503 14:13:51.796249 13530 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 633. Preceding OpId from leader: term: 1 index: 634. (index mismatch)
I20240503 14:13:51.797046 17479 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 634, Last known committed idx: 633, Time since last communication: 0.000s
I20240503 14:13:51.798166 12878 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 633. Preceding OpId from leader: term: 1 index: 634. (index mismatch)
I20240503 14:13:51.798138 17255 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 634, Last known committed idx: 633, Time since last communication: 0.000s
I20240503 14:13:51.798460 13273 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 633. Preceding OpId from leader: term: 1 index: 634. (index mismatch)
I20240503 14:13:51.799659 17255 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 634, Last known committed idx: 633, Time since last communication: 0.000s
I20240503 14:13:51.800274 17255 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 634, Last known committed idx: 633, Time since last communication: 0.000s
I20240503 14:13:51.816270 17255 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.634: config changed from index 296 to 634, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.821350 13269 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.634: config changed from index 296 to 634, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.829624 12738 catalog_manager.cc:5484] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index 296 to 634, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.829849 13530 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.634: config changed from index 296 to 634, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.834879 12878 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.634: config changed from index 296 to 634, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.838289 14502 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.634: config changed from index 296 to 634, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
W20240503 14:13:51.859498 13078 consensus_peers.cc:485] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 -> Peer 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135): Couldn't send request to peer 6a7786296dae40d4aa2c86869ddfb919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e130a7b478f4f91af11bef82ebf7236. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:51.881373 13142 consensus_queue.cc:237] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 503, Committed index: 503, Last appended: 1.503, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
W20240503 14:13:51.884864 13078 consensus_peers.cc:485] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 -> Peer 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135): Couldn't send request to peer 6a7786296dae40d4aa2c86869ddfb919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d17c42814e24b70b25c24ea15a336f3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:51.885185 14502 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 503. Preceding OpId from leader: term: 1 index: 504. (index mismatch)
I20240503 14:13:51.885959 17516 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 504, Last known committed idx: 503, Time since last communication: 0.000s
I20240503 14:13:51.886149 13010 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 503. Preceding OpId from leader: term: 1 index: 504. (index mismatch)
I20240503 14:13:51.886168 12878 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 503. Preceding OpId from leader: term: 1 index: 504. (index mismatch)
I20240503 14:13:51.886829 13399 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 503. Preceding OpId from leader: term: 1 index: 504. (index mismatch)
I20240503 14:13:51.887385 17479 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 504, Last known committed idx: 503, Time since last communication: 0.000s
I20240503 14:13:51.887897 17515 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 504, Last known committed idx: 503, Time since last communication: 0.000s
I20240503 14:13:51.888211 17479 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 504, Last known committed idx: 503, Time since last communication: 0.000s
I20240503 14:13:51.894666 17516 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.504: config changed from index 162 to 504, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.898193 13399 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.504: config changed from index 162 to 504, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.900405 14502 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.504: config changed from index 162 to 504, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.902624 12738 catalog_manager.cc:5484] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: config changed from index 162 to 504, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:51.911300 13010 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.504: config changed from index 162 to 504, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:51.911649 12878 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.504: config changed from index 162 to 504, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.018936 13010 consensus_queue.cc:237] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:52.024369 13530 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:52.025183 17563 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:52.026599 14502 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:52.027796 13399 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:52.028831 17211 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:52.029088 13273 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:52.029402 17343 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:52.032543 17343 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:52.034035 17343 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.034600 13399 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.042222 13273 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
W20240503 14:13:52.044342 12948 consensus_peers.cc:485] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 -> Peer 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135): Couldn't send request to peer 6a7786296dae40d4aa2c86869ddfb919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e82229abb5be4ed1abc3fa200671416e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.044236 12742 catalog_manager.cc:5484] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.054894 13530 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.063114 14506 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.090430 13399 consensus_queue.cc:237] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 734, Committed index: 734, Last appended: 1.734, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 735 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
W20240503 14:13:52.094637 13338 consensus_peers.cc:485] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 -> Peer 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135): Couldn't send request to peer 6a7786296dae40d4aa2c86869ddfb919. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a959a6c207af45009ff7807ea6f20ee2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:52.095643 14502 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 734. Preceding OpId from leader: term: 1 index: 735. (index mismatch)
I20240503 14:13:52.095863 13273 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 734. Preceding OpId from leader: term: 1 index: 735. (index mismatch)
I20240503 14:13:52.096660 17554 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 735, Last known committed idx: 734, Time since last communication: 0.000s
I20240503 14:13:52.096983 17413 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 735, Last known committed idx: 734, Time since last communication: 0.000s
I20240503 14:13:52.098856 12878 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 734. Preceding OpId from leader: term: 1 index: 735. (index mismatch)
I20240503 14:13:52.101033 17554 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 735, Last known committed idx: 734, Time since last communication: 0.000s
I20240503 14:13:52.125803 17413 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.735: config changed from index 258 to 735, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 735 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.130868 13530 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 734. Preceding OpId from leader: term: 1 index: 735. (index mismatch)
I20240503 14:13:52.130718 12877 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.735: config changed from index 258 to 735, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 735 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.132138 17554 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 735, Last known committed idx: 734, Time since last communication: 0.000s
I20240503 14:13:52.132069 13273 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.735: config changed from index 258 to 735, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 735 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.138016 14502 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.735: config changed from index 258 to 735, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 735 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.138392 12738 catalog_manager.cc:5484] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 reported cstate change: config changed from index 258 to 735, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 735 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.147927 13530 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.735: config changed from index 258 to 735, NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) added. New config: { opid_index: 735 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } } }
I20240503 14:13:52.255555 17485 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap replayed 1/1 log segments. Stats: ops{read=384 overwritten=0 applied=384 ignored=0} inserts{seen=1012 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:52.255967 17485 tablet_bootstrap.cc:492] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap complete.
I20240503 14:13:52.256243 17485 ts_tablet_manager.cc:1367] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919: Time spent bootstrapping tablet: real 0.481s user 0.385s sys 0.028s
I20240503 14:13:52.256959 17485 raft_consensus.cc:348] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:52.257192 17485 raft_consensus.cc:729] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a7786296dae40d4aa2c86869ddfb919, State: Initialized, Role: LEARNER
I20240503 14:13:52.257460 17485 consensus_queue.cc:260] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 384, Last appended: 1.384, Last appended by leader: 384, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:52.258308 17485 ts_tablet_manager.cc:1398] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:13:52.259397 13554 tablet_copy_service.cc:342] P af88d71da3a54f779cf4f651f4393bb6: Request end of tablet copy session 6a7786296dae40d4aa2c86869ddfb919-f99a260ae30a4efda1126bf96b1ac02f received from {username='slave'} at 127.10.145.8:35377
I20240503 14:13:52.259537 13554 tablet_copy_service.cc:433] P af88d71da3a54f779cf4f651f4393bb6: ending tablet copy session 6a7786296dae40d4aa2c86869ddfb919-f99a260ae30a4efda1126bf96b1ac02f on tablet f99a260ae30a4efda1126bf96b1ac02f with peer 6a7786296dae40d4aa2c86869ddfb919
I20240503 14:13:52.368649 17485 ts_tablet_manager.cc:902] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919: Initiating tablet copy from peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
I20240503 14:13:52.370035 17485 tablet_copy_client.cc:308] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:41101
I20240503 14:13:52.376384 13164 tablet_copy_service.cc:140] P 7b75d8c9e35a416ba24798fde8b9def0: Received BeginTabletCopySession request for tablet 3d17c42814e24b70b25c24ea15a336f3 from peer 6a7786296dae40d4aa2c86869ddfb919 ({username='slave'} at 127.10.145.8:41223)
I20240503 14:13:52.376559 13164 tablet_copy_service.cc:161] P 7b75d8c9e35a416ba24798fde8b9def0: Beginning new tablet copy session on tablet 3d17c42814e24b70b25c24ea15a336f3 from peer 6a7786296dae40d4aa2c86869ddfb919 at {username='slave'} at 127.10.145.8:41223: session id = 6a7786296dae40d4aa2c86869ddfb919-3d17c42814e24b70b25c24ea15a336f3
I20240503 14:13:52.378705 13164 tablet_copy_source_session.cc:205] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:52.379977 17485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d17c42814e24b70b25c24ea15a336f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.384799 17485 tablet_copy_client.cc:783] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:52.385083 17485 tablet_copy_client.cc:647] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:52.388147 17485 tablet_copy_client.cc:523] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:52.389981 17613 ts_tablet_manager.cc:902] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919: Initiating tablet copy from peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
I20240503 14:13:52.391592 17613 tablet_copy_client.cc:308] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:41101
I20240503 14:13:52.392130 17485 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap starting.
I20240503 14:13:52.392463 13164 tablet_copy_service.cc:140] P 7b75d8c9e35a416ba24798fde8b9def0: Received BeginTabletCopySession request for tablet 5e130a7b478f4f91af11bef82ebf7236 from peer 6a7786296dae40d4aa2c86869ddfb919 ({username='slave'} at 127.10.145.8:41223)
I20240503 14:13:52.392644 13164 tablet_copy_service.cc:161] P 7b75d8c9e35a416ba24798fde8b9def0: Beginning new tablet copy session on tablet 5e130a7b478f4f91af11bef82ebf7236 from peer 6a7786296dae40d4aa2c86869ddfb919 at {username='slave'} at 127.10.145.8:41223: session id = 6a7786296dae40d4aa2c86869ddfb919-5e130a7b478f4f91af11bef82ebf7236
I20240503 14:13:52.395641 13164 tablet_copy_source_session.cc:205] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:52.397197 17613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e130a7b478f4f91af11bef82ebf7236. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.401859 17618 ts_tablet_manager.cc:902] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919: Initiating tablet copy from peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
I20240503 14:13:52.402973 17613 tablet_copy_client.cc:783] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:52.403285 17613 tablet_copy_client.cc:647] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:52.407426 17613 tablet_copy_client.cc:523] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:52.412166 17613 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap starting.
I20240503 14:13:52.419864 17618 tablet_copy_client.cc:308] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:36441
I20240503 14:13:52.428580 13034 tablet_copy_service.cc:140] P 3678714622584fa4883e4bf07fcabd75: Received BeginTabletCopySession request for tablet e82229abb5be4ed1abc3fa200671416e from peer 6a7786296dae40d4aa2c86869ddfb919 ({username='slave'} at 127.10.145.8:33653)
I20240503 14:13:52.428795 13034 tablet_copy_service.cc:161] P 3678714622584fa4883e4bf07fcabd75: Beginning new tablet copy session on tablet e82229abb5be4ed1abc3fa200671416e from peer 6a7786296dae40d4aa2c86869ddfb919 at {username='slave'} at 127.10.145.8:33653: session id = 6a7786296dae40d4aa2c86869ddfb919-e82229abb5be4ed1abc3fa200671416e
I20240503 14:13:52.431845 13034 tablet_copy_source_session.cc:205] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:52.433760 17618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82229abb5be4ed1abc3fa200671416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.442385 17618 tablet_copy_client.cc:783] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:52.442821 17618 tablet_copy_client.cc:647] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:52.447454 17618 tablet_copy_client.cc:523] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:52.453181 17618 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap starting.
I20240503 14:13:52.479133 17618 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:52.480257 17618 tablet_bootstrap.cc:492] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap complete.
I20240503 14:13:52.480602 17618 ts_tablet_manager.cc:1367] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.007s
I20240503 14:13:52.481729 17618 raft_consensus.cc:348] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:52.482143 17618 raft_consensus.cc:729] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a7786296dae40d4aa2c86869ddfb919, State: Initialized, Role: LEARNER
I20240503 14:13:52.482421 17618 consensus_queue.cc:260] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:52.483312 17618 ts_tablet_manager.cc:1398] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:52.484540 13034 tablet_copy_service.cc:342] P 3678714622584fa4883e4bf07fcabd75: Request end of tablet copy session 6a7786296dae40d4aa2c86869ddfb919-e82229abb5be4ed1abc3fa200671416e received from {username='slave'} at 127.10.145.8:33653
I20240503 14:13:52.484683 13034 tablet_copy_service.cc:433] P 3678714622584fa4883e4bf07fcabd75: ending tablet copy session 6a7786296dae40d4aa2c86869ddfb919-e82229abb5be4ed1abc3fa200671416e on tablet e82229abb5be4ed1abc3fa200671416e with peer 6a7786296dae40d4aa2c86869ddfb919
I20240503 14:13:52.632000 16978 raft_consensus.cc:1203] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.383->[1.384-1.384] Dedup: 1.384->[]
I20240503 14:13:52.634626 17618 ts_tablet_manager.cc:902] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919: Initiating tablet copy from peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005)
I20240503 14:13:52.647294 17618 tablet_copy_client.cc:308] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.5:38005
I20240503 14:13:52.648157 13424 tablet_copy_service.cc:140] P e148d8a3757340ab9288ea98cabb5ad3: Received BeginTabletCopySession request for tablet a959a6c207af45009ff7807ea6f20ee2 from peer 6a7786296dae40d4aa2c86869ddfb919 ({username='slave'} at 127.10.145.8:40893)
I20240503 14:13:52.648331 13424 tablet_copy_service.cc:161] P e148d8a3757340ab9288ea98cabb5ad3: Beginning new tablet copy session on tablet a959a6c207af45009ff7807ea6f20ee2 from peer 6a7786296dae40d4aa2c86869ddfb919 at {username='slave'} at 127.10.145.8:40893: session id = 6a7786296dae40d4aa2c86869ddfb919-a959a6c207af45009ff7807ea6f20ee2
I20240503 14:13:52.651288 13424 tablet_copy_source_session.cc:205] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:52.655313 17618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a959a6c207af45009ff7807ea6f20ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.663228 17618 tablet_copy_client.cc:783] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:52.663612 17618 tablet_copy_client.cc:647] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:52.671036 17618 tablet_copy_client.cc:523] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:52.675995 17618 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap starting.
I20240503 14:13:52.966727 16978 raft_consensus.cc:1203] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:53.137790 17485 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap replayed 1/1 log segments. Stats: ops{read=504 overwritten=0 applied=504 ignored=0} inserts{seen=1132 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.138377 17485 tablet_bootstrap.cc:492] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap complete.
I20240503 14:13:53.138806 17485 ts_tablet_manager.cc:1367] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919: Time spent bootstrapping tablet: real 0.747s user 0.564s sys 0.045s
I20240503 14:13:53.139803 17485 raft_consensus.cc:348] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:53.140125 17485 raft_consensus.cc:729] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a7786296dae40d4aa2c86869ddfb919, State: Initialized, Role: LEARNER
I20240503 14:13:53.140547 17485 consensus_queue.cc:260] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 504, Last appended: 1.504, Last appended by leader: 504, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:53.141229 17485 ts_tablet_manager.cc:1398] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:53.142271 13164 tablet_copy_service.cc:342] P 7b75d8c9e35a416ba24798fde8b9def0: Request end of tablet copy session 6a7786296dae40d4aa2c86869ddfb919-3d17c42814e24b70b25c24ea15a336f3 received from {username='slave'} at 127.10.145.8:41223
I20240503 14:13:53.142423 13164 tablet_copy_service.cc:433] P 7b75d8c9e35a416ba24798fde8b9def0: ending tablet copy session 6a7786296dae40d4aa2c86869ddfb919-3d17c42814e24b70b25c24ea15a336f3 on tablet 3d17c42814e24b70b25c24ea15a336f3 with peer 6a7786296dae40d4aa2c86869ddfb919
I20240503 14:13:53.176100 17304 raft_consensus.cc:1050] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6: attempting to promote NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 to VOTER
I20240503 14:13:53.179544 17304 consensus_queue.cc:237] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 384, Committed index: 384, Last appended: 1.384, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:53.186117 16978 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 384. Preceding OpId from leader: term: 1 index: 385. (index mismatch)
I20240503 14:13:53.186379 12878 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 384. Preceding OpId from leader: term: 1 index: 385. (index mismatch)
I20240503 14:13:53.187637 13399 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 384. Preceding OpId from leader: term: 1 index: 385. (index mismatch)
I20240503 14:13:53.188220 17651 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I20240503 14:13:53.188895 17651 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I20240503 14:13:53.202975 13010 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 384. Preceding OpId from leader: term: 1 index: 385. (index mismatch)
I20240503 14:13:53.203898 17688 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I20240503 14:13:53.206782 13273 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 384. Preceding OpId from leader: term: 1 index: 385. (index mismatch)
I20240503 14:13:53.211300 17651 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I20240503 14:13:53.213129 17651 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I20240503 14:13:53.271320 17651 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.385: config changed from index 384 to 385, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.277112 12878 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.385: config changed from index 384 to 385, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.287868 12735 catalog_manager.cc:5484] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 384 to 385, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:53.289031 13398 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.385: config changed from index 384 to 385, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.298035 13273 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.385: config changed from index 384 to 385, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.313794 13014 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.385: config changed from index 384 to 385, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.323760 16978 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.385: config changed from index 384 to 385, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.358688 17613 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap replayed 1/1 log segments. Stats: ops{read=634 overwritten=0 applied=634 ignored=0} inserts{seen=1316 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.359337 17613 tablet_bootstrap.cc:492] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap complete.
I20240503 14:13:53.359817 17613 ts_tablet_manager.cc:1367] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919: Time spent bootstrapping tablet: real 0.948s user 0.681s sys 0.087s
I20240503 14:13:53.361084 17613 raft_consensus.cc:348] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:53.361464 17613 raft_consensus.cc:729] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a7786296dae40d4aa2c86869ddfb919, State: Initialized, Role: LEARNER
I20240503 14:13:53.361795 17613 consensus_queue.cc:260] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 634, Last appended: 1.634, Last appended by leader: 634, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:53.362913 17613 ts_tablet_manager.cc:1398] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:53.364430 13164 tablet_copy_service.cc:342] P 7b75d8c9e35a416ba24798fde8b9def0: Request end of tablet copy session 6a7786296dae40d4aa2c86869ddfb919-5e130a7b478f4f91af11bef82ebf7236 received from {username='slave'} at 127.10.145.8:41223
I20240503 14:13:53.364578 13164 tablet_copy_service.cc:433] P 7b75d8c9e35a416ba24798fde8b9def0: ending tablet copy session 6a7786296dae40d4aa2c86869ddfb919-5e130a7b478f4f91af11bef82ebf7236 on tablet 5e130a7b478f4f91af11bef82ebf7236 with peer 6a7786296dae40d4aa2c86869ddfb919
I20240503 14:13:53.364535 13530 consensus_queue.cc:237] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 385, Committed index: 385, Last appended: 1.385, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:53.367751 16978 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240503 14:13:53.367753 13014 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240503 14:13:53.368402 17688 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20240503 14:13:53.369693 12878 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240503 14:13:53.370150 13273 raft_consensus.cc:1261] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer af88d71da3a54f779cf4f651f4393bb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240503 14:13:53.370929 17304 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20240503 14:13:53.371634 17651 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20240503 14:13:53.372354 17688 consensus_queue.cc:1035] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20240503 14:13:53.374198 17688 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 [term 1 LEADER]: Committing config change with OpId 1.386: config changed from index 385 to 386, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.378095 13273 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index 385 to 386, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.380729 12729 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f99a260ae30a4efda1126bf96b1ac02f with cas_config_opid_index 385: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:53.381831 16978 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index 385 to 386, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.383275 13010 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index 385 to 386, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.384431 12737 catalog_manager.cc:5484] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 385 to 386, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New cstate: current_term: 1 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:53.377010 12878 raft_consensus.cc:2932] T f99a260ae30a4efda1126bf96b1ac02f P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index 385 to 386, VOTER e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5) evicted. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.389165 13382 tablet_service.cc:1514] Processing DeleteTablet for tablet f99a260ae30a4efda1126bf96b1ac02f with delete_type TABLET_DATA_TOMBSTONED (TS e148d8a3757340ab9288ea98cabb5ad3 not found in new config with opid_index 386) from {username='slave'} at 127.0.0.1:52634
I20240503 14:13:53.389883 17707 tablet_replica.cc:330] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3: stopping tablet replica
I20240503 14:13:53.390097 17707 raft_consensus.cc:2229] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:53.390465 17707 raft_consensus.cc:2258] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:53.394934 17707 ts_tablet_manager.cc:1875] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:53.404784 17707 ts_tablet_manager.cc:1888] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.385
I20240503 14:13:53.407034 17707 log.cc:1196] T f99a260ae30a4efda1126bf96b1ac02f P e148d8a3757340ab9288ea98cabb5ad3: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/wals/f99a260ae30a4efda1126bf96b1ac02f
I20240503 14:13:53.407955 12727 catalog_manager.cc:4830] TS e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): tablet f99a260ae30a4efda1126bf96b1ac02f (table rebalance_test_table_3 [id=dcece03e519d431baaf2120b21910f9a]) successfully deleted
I20240503 14:13:53.521903 16978 raft_consensus.cc:1203] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.503->[1.504-1.504] Dedup: 1.504->[]
I20240503 14:13:53.547317 17674 raft_consensus.cc:1050] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75: attempting to promote NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 to VOTER
I20240503 14:13:53.548535 17674 consensus_queue.cc:237] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:53.551913 16978 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:53.552006 14506 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:53.552713 17568 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:53.553018 17675 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:53.554620 13399 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:53.558609 17568 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:53.569797 13530 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:53.570863 13269 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:53.571077 17751 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:53.571852 17675 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:53.576653 13530 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.576634 13399 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.574599 17675 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.579084 13269 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.583118 12734 catalog_manager.cc:5484] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 2 to 3, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.590087 16978 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.592613 14506 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.598795 13010 consensus_queue.cc:237] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:53.602222 13530 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.603297 16977 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.603801 17675 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.604926 13399 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.604982 17675 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.606926 17618 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap replayed 1/1 log segments. Stats: ops{read=735 overwritten=0 applied=735 ignored=0} inserts{seen=1404 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.606911 14502 raft_consensus.cc:1261] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 3678714622584fa4883e4bf07fcabd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.607338 17675 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.608639 17618 tablet_bootstrap.cc:492] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919: Bootstrap complete.
I20240503 14:13:53.609073 17618 ts_tablet_manager.cc:1367] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919: Time spent bootstrapping tablet: real 0.933s user 0.774s sys 0.068s
I20240503 14:13:53.609212 17675 consensus_queue.cc:1035] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.609990 17618 raft_consensus.cc:348] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 735 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:53.610280 17618 raft_consensus.cc:729] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a7786296dae40d4aa2c86869ddfb919, State: Initialized, Role: LEARNER
I20240503 14:13:53.610632 16977 raft_consensus.cc:1203] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.633->[1.634-1.634] Dedup: 1.634->[]
I20240503 14:13:53.610535 17618 consensus_queue.cc:260] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 735, Last appended: 1.735, Last appended by leader: 735, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 735 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: true } }
I20240503 14:13:53.612195 17674 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P 3678714622584fa4883e4bf07fcabd75 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.613759 13532 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.614207 17618 ts_tablet_manager.cc:1398] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:53.615015 13424 tablet_copy_service.cc:342] P e148d8a3757340ab9288ea98cabb5ad3: Request end of tablet copy session 6a7786296dae40d4aa2c86869ddfb919-a959a6c207af45009ff7807ea6f20ee2 received from {username='slave'} at 127.10.145.8:40893
I20240503 14:13:53.615134 13424 tablet_copy_service.cc:433] P e148d8a3757340ab9288ea98cabb5ad3: ending tablet copy session 6a7786296dae40d4aa2c86869ddfb919-a959a6c207af45009ff7807ea6f20ee2 on tablet a959a6c207af45009ff7807ea6f20ee2 with peer 6a7786296dae40d4aa2c86869ddfb919
I20240503 14:13:53.617105 13399 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.617851 14502 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.621524 12726 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e82229abb5be4ed1abc3fa200671416e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:53.622336 16977 raft_consensus.cc:2932] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.622699 12737 catalog_manager.cc:5484] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 3 to 4, VOTER 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "3678714622584fa4883e4bf07fcabd75" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.627249 13254 tablet_service.cc:1514] Processing DeleteTablet for tablet e82229abb5be4ed1abc3fa200671416e with delete_type TABLET_DATA_TOMBSTONED (TS 2cd2acf7d8274b5a95482dffd0823a08 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56270
I20240503 14:13:53.629299 17755 tablet_replica.cc:330] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: stopping tablet replica
I20240503 14:13:53.629529 17755 raft_consensus.cc:2229] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:53.629859 17755 raft_consensus.cc:2258] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:53.631902 17755 ts_tablet_manager.cc:1875] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:53.637202 17755 ts_tablet_manager.cc:1888] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:53.637362 17755 log.cc:1196] T e82229abb5be4ed1abc3fa200671416e P 2cd2acf7d8274b5a95482dffd0823a08: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/e82229abb5be4ed1abc3fa200671416e
I20240503 14:13:53.638136 12727 catalog_manager.cc:4830] TS 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): tablet e82229abb5be4ed1abc3fa200671416e (table TestTable [id=ed6a4b9d46264728bfc80066eea83360]) successfully deleted
I20240503 14:13:53.863680 16977 raft_consensus.cc:1203] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.734->[1.735-1.735] Dedup: 1.735->[]
I20240503 14:13:53.935006 17668 raft_consensus.cc:1050] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0: attempting to promote NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 to VOTER
I20240503 14:13:53.936161 17668 consensus_queue.cc:237] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 504, Committed index: 504, Last appended: 1.504, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:53.939131 13399 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 504. Preceding OpId from leader: term: 1 index: 505. (index mismatch)
I20240503 14:13:53.939147 14502 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 504. Preceding OpId from leader: term: 1 index: 505. (index mismatch)
I20240503 14:13:53.939919 16977 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 504. Preceding OpId from leader: term: 1 index: 505. (index mismatch)
I20240503 14:13:53.939831 17746 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 505, Last known committed idx: 504, Time since last communication: 0.000s
I20240503 14:13:53.940500 17746 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 505, Last known committed idx: 504, Time since last communication: 0.000s
I20240503 14:13:53.941603 13010 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 504. Preceding OpId from leader: term: 1 index: 505. (index mismatch)
I20240503 14:13:53.941601 17515 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 505, Last known committed idx: 504, Time since last communication: 0.000s
I20240503 14:13:53.942724 17793 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 505, Last known committed idx: 504, Time since last communication: 0.000s
I20240503 14:13:53.945257 12878 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 504. Preceding OpId from leader: term: 1 index: 505. (index mismatch)
I20240503 14:13:53.945953 17746 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 505, Last known committed idx: 504, Time since last communication: 0.000s
I20240503 14:13:53.948987 17668 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.505: config changed from index 504 to 505, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.950182 12878 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.505: config changed from index 504 to 505, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.953908 16977 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.505: config changed from index 504 to 505, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.953878 14502 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.505: config changed from index 504 to 505, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.958671 12734 catalog_manager.cc:5484] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 reported cstate change: config changed from index 504 to 505, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.962180 13399 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.505: config changed from index 504 to 505, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.971364 13010 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Committing config change with OpId 1.505: config changed from index 504 to 505, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 505 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } attrs { replace: true } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.972260 13138 consensus_queue.cc:237] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 505, Committed index: 505, Last appended: 1.505, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:53.976570 12878 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 505. Preceding OpId from leader: term: 1 index: 506. (index mismatch)
I20240503 14:13:53.977522 17800 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 506, Last known committed idx: 505, Time since last communication: 0.000s
I20240503 14:13:53.978049 14502 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 505. Preceding OpId from leader: term: 1 index: 506. (index mismatch)
I20240503 14:13:53.978907 17800 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 506, Last known committed idx: 505, Time since last communication: 0.000s
I20240503 14:13:53.979437 16977 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 505. Preceding OpId from leader: term: 1 index: 506. (index mismatch)
I20240503 14:13:53.979429 13399 raft_consensus.cc:1261] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 505. Preceding OpId from leader: term: 1 index: 506. (index mismatch)
I20240503 14:13:53.980623 17799 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 506, Last known committed idx: 505, Time since last communication: 0.000s
I20240503 14:13:53.982077 17800 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.506: config changed from index 505 to 506, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.982692 14502 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.506: config changed from index 505 to 506, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.986775 17515 consensus_queue.cc:1035] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 506, Last known committed idx: 505, Time since last communication: 0.000s
I20240503 14:13:53.987488 12728 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d17c42814e24b70b25c24ea15a336f3 with cas_config_opid_index 505: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:53.988934 12737 catalog_manager.cc:5484] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de reported cstate change: config changed from index 505 to 506, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.989858 13399 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Committing config change with OpId 1.506: config changed from index 505 to 506, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.990695 12878 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.506: config changed from index 505 to 506, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:53.991241 16977 raft_consensus.cc:2932] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.506: config changed from index 505 to 506, VOTER 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) evicted. New config: { opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.005448 12992 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d17c42814e24b70b25c24ea15a336f3 with delete_type TABLET_DATA_TOMBSTONED (TS 3678714622584fa4883e4bf07fcabd75 not found in new config with opid_index 506) from {username='slave'} at 127.0.0.1:55928
I20240503 14:13:54.009733 17807 tablet_replica.cc:330] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75: stopping tablet replica
I20240503 14:13:54.009974 17807 raft_consensus.cc:2229] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:54.010814 17807 raft_consensus.cc:2258] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:54.012568 17807 ts_tablet_manager.cc:1875] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:54.020272 17807 ts_tablet_manager.cc:1888] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.505
I20240503 14:13:54.020432 17807 log.cc:1196] T 3d17c42814e24b70b25c24ea15a336f3 P 3678714622584fa4883e4bf07fcabd75: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/3d17c42814e24b70b25c24ea15a336f3
I20240503 14:13:54.022603 12726 catalog_manager.cc:4830] TS 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): tablet 3d17c42814e24b70b25c24ea15a336f3 (table rebalance_test_table_4 [id=326309dc584e45b782a9eb56bd9d5b93]) successfully deleted
I20240503 14:13:54.158772 17515 raft_consensus.cc:1050] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0: attempting to promote NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 to VOTER
I20240503 14:13:54.159955 17515 consensus_queue.cc:237] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 634, Committed index: 634, Last appended: 1.634, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 635 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:54.162664 14502 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 634. Preceding OpId from leader: term: 1 index: 635. (index mismatch)
I20240503 14:13:54.162714 12878 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 634. Preceding OpId from leader: term: 1 index: 635. (index mismatch)
I20240503 14:13:54.162767 13530 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 634. Preceding OpId from leader: term: 1 index: 635. (index mismatch)
I20240503 14:13:54.163280 16977 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 634. Preceding OpId from leader: term: 1 index: 635. (index mismatch)
I20240503 14:13:54.163302 17793 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 635, Last known committed idx: 634, Time since last communication: 0.000s
I20240503 14:13:54.163741 17800 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 635, Last known committed idx: 634, Time since last communication: 0.000s
I20240503 14:13:54.164278 17800 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 635, Last known committed idx: 634, Time since last communication: 0.000s
I20240503 14:13:54.164359 13273 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 634. Preceding OpId from leader: term: 1 index: 635. (index mismatch)
I20240503 14:13:54.164654 17515 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 635, Last known committed idx: 634, Time since last communication: 0.000s
I20240503 14:13:54.165450 17793 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 635, Last known committed idx: 634, Time since last communication: 0.000s
I20240503 14:13:54.179385 17515 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.635: config changed from index 634 to 635, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 635 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.181394 13530 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.635: config changed from index 634 to 635, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 635 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.181368 13273 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.635: config changed from index 634 to 635, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 635 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.185334 12878 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.635: config changed from index 634 to 635, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 635 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.185312 16977 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.635: config changed from index 634 to 635, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 635 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.186437 12734 catalog_manager.cc:5484] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 reported cstate change: config changed from index 634 to 635, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 635 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:54.187333 14502 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.635: config changed from index 634 to 635, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 635 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.198508 13138 consensus_queue.cc:237] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 635, Committed index: 635, Last appended: 1.635, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 636 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:54.200531 13530 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 635. Preceding OpId from leader: term: 1 index: 636. (index mismatch)
I20240503 14:13:54.200682 13273 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 635. Preceding OpId from leader: term: 1 index: 636. (index mismatch)
I20240503 14:13:54.200896 12878 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 635. Preceding OpId from leader: term: 1 index: 636. (index mismatch)
I20240503 14:13:54.201220 17800 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 636, Last known committed idx: 635, Time since last communication: 0.000s
I20240503 14:13:54.201539 16977 raft_consensus.cc:1261] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Refusing update from remote peer 7b75d8c9e35a416ba24798fde8b9def0: Log matching property violated. Preceding OpId in replica: term: 1 index: 635. Preceding OpId from leader: term: 1 index: 636. (index mismatch)
I20240503 14:13:54.201632 17801 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 636, Last known committed idx: 635, Time since last communication: 0.000s
I20240503 14:13:54.202081 17793 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 636, Last known committed idx: 635, Time since last communication: 0.000s
I20240503 14:13:54.202378 17515 consensus_queue.cc:1035] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 636, Last known committed idx: 635, Time since last communication: 0.000s
I20240503 14:13:54.204921 17793 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 LEADER]: Committing config change with OpId 1.636: config changed from index 635 to 636, VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) evicted. New config: { opid_index: 636 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.206135 12878 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.636: config changed from index 635 to 636, VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) evicted. New config: { opid_index: 636 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.206172 16977 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.636: config changed from index 635 to 636, VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) evicted. New config: { opid_index: 636 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.206324 13530 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.636: config changed from index 635 to 636, VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) evicted. New config: { opid_index: 636 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.208966 12728 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e130a7b478f4f91af11bef82ebf7236 with cas_config_opid_index 635: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:54.212407 12733 catalog_manager.cc:5484] T 5e130a7b478f4f91af11bef82ebf7236 P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: config changed from index 635 to 636, VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) evicted. New cstate: current_term: 1 leader_uuid: "7b75d8c9e35a416ba24798fde8b9def0" committed_config { opid_index: 636 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.212502 13273 raft_consensus.cc:2932] T 5e130a7b478f4f91af11bef82ebf7236 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.636: config changed from index 635 to 636, VOTER 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7) evicted. New config: { opid_index: 636 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.224784 14485 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e130a7b478f4f91af11bef82ebf7236 with delete_type TABLET_DATA_TOMBSTONED (TS 21d8b64a50ba4c919cb9e711d18fb7de not found in new config with opid_index 636) from {username='slave'} at 127.0.0.1:36038
I20240503 14:13:54.225771 17837 tablet_replica.cc:330] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: stopping tablet replica
I20240503 14:13:54.226294 17837 raft_consensus.cc:2229] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:54.226706 17837 raft_consensus.cc:2258] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:54.229771 17837 ts_tablet_manager.cc:1875] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:54.237656 17837 ts_tablet_manager.cc:1888] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.635
I20240503 14:13:54.237857 17837 log.cc:1196] T 5e130a7b478f4f91af11bef82ebf7236 P 21d8b64a50ba4c919cb9e711d18fb7de: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/wals/5e130a7b478f4f91af11bef82ebf7236
I20240503 14:13:54.238790 12727 catalog_manager.cc:4830] TS 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): tablet 5e130a7b478f4f91af11bef82ebf7236 (table rebalance_test_table_1 [id=efd4bd07f51242c7a453d29157fb1cc1]) successfully deleted
I20240503 14:13:54.322439 17854 raft_consensus.cc:1050] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3: attempting to promote NON_VOTER 6a7786296dae40d4aa2c86869ddfb919 to VOTER
I20240503 14:13:54.323294 17854 consensus_queue.cc:237] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 735, Committed index: 735, Last appended: 1.735, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:54.326161 14502 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 735. Preceding OpId from leader: term: 1 index: 736. (index mismatch)
I20240503 14:13:54.326221 13273 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 735. Preceding OpId from leader: term: 1 index: 736. (index mismatch)
I20240503 14:13:54.326328 16977 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 LEARNER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 735. Preceding OpId from leader: term: 1 index: 736. (index mismatch)
I20240503 14:13:54.327034 17575 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 736, Last known committed idx: 735, Time since last communication: 0.000s
I20240503 14:13:54.327383 17853 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 736, Last known committed idx: 735, Time since last communication: 0.000s
I20240503 14:13:54.327941 17575 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 736, Last known committed idx: 735, Time since last communication: 0.000s
I20240503 14:13:54.333904 17575 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.736: config changed from index 735 to 736, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.338178 16977 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.736: config changed from index 735 to 736, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.345435 12734 catalog_manager.cc:5484] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919 reported cstate change: config changed from index 735 to 736, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.347718 13530 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 735. Preceding OpId from leader: term: 1 index: 736. (index mismatch)
I20240503 14:13:54.348376 17853 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 736, Last known committed idx: 735, Time since last communication: 0.000s
I20240503 14:13:54.348654 12878 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 735. Preceding OpId from leader: term: 1 index: 736. (index mismatch)
I20240503 14:13:54.350284 13530 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Committing config change with OpId 1.736: config changed from index 735 to 736, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.352552 17854 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 736, Last known committed idx: 735, Time since last communication: 0.000s
I20240503 14:13:54.354651 14502 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.736: config changed from index 735 to 736, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.355029 13273 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.736: config changed from index 735 to 736, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.361228 12878 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.736: config changed from index 735 to 736, 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 736 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } attrs { replace: true } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.964025 13399 consensus_queue.cc:237] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 736, Committed index: 736, Last appended: 1.736, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:13:54.966575 12878 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 736. Preceding OpId from leader: term: 1 index: 737. (index mismatch)
I20240503 14:13:54.966590 14502 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 736. Preceding OpId from leader: term: 1 index: 737. (index mismatch)
I20240503 14:13:54.966689 16977 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 736. Preceding OpId from leader: term: 1 index: 737. (index mismatch)
I20240503 14:13:54.966773 13273 raft_consensus.cc:1261] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Refusing update from remote peer e148d8a3757340ab9288ea98cabb5ad3: Log matching property violated. Preceding OpId in replica: term: 1 index: 736. Preceding OpId from leader: term: 1 index: 737. (index mismatch)
I20240503 14:13:54.967329 17880 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 737, Last known committed idx: 736, Time since last communication: 0.000s
I20240503 14:13:54.967667 17860 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 737, Last known committed idx: 736, Time since last communication: 0.000s
I20240503 14:13:54.967945 17873 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 737, Last known committed idx: 736, Time since last communication: 0.000s
I20240503 14:13:54.968201 17874 consensus_queue.cc:1035] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 737, Last known committed idx: 736, Time since last communication: 0.000s
I20240503 14:13:54.971930 17860 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 LEADER]: Committing config change with OpId 1.737: config changed from index 736 to 737, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.972682 14502 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Committing config change with OpId 1.737: config changed from index 736 to 737, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.977053 13273 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Committing config change with OpId 1.737: config changed from index 736 to 737, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.977317 12734 catalog_manager.cc:5484] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 reported cstate change: config changed from index 736 to 737, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "e148d8a3757340ab9288ea98cabb5ad3" committed_config { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:54.978741 16977 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Committing config change with OpId 1.737: config changed from index 736 to 737, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:54.979245 12727 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a959a6c207af45009ff7807ea6f20ee2 with cas_config_opid_index 736: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:54.984660 12878 raft_consensus.cc:2932] T a959a6c207af45009ff7807ea6f20ee2 P 72b07e9b53b9471989185277045db8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.737: config changed from index 736 to 737, VOTER af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6) evicted. New config: { opid_index: 737 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } } }
I20240503 14:13:55.000414 13514 tablet_service.cc:1514] Processing DeleteTablet for tablet a959a6c207af45009ff7807ea6f20ee2 with delete_type TABLET_DATA_TOMBSTONED (TS af88d71da3a54f779cf4f651f4393bb6 not found in new config with opid_index 737) from {username='slave'} at 127.0.0.1:48902
I20240503 14:13:55.001080 17886 tablet_replica.cc:330] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6: stopping tablet replica
I20240503 14:13:55.001267 17886 raft_consensus.cc:2229] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.001533 17886 raft_consensus.cc:2258] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.003494 17886 ts_tablet_manager.cc:1875] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.008258 17886 ts_tablet_manager.cc:1888] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.736
I20240503 14:13:55.008411 17886 log.cc:1196] T a959a6c207af45009ff7807ea6f20ee2 P af88d71da3a54f779cf4f651f4393bb6: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/a959a6c207af45009ff7807ea6f20ee2
I20240503 14:13:55.009385 12729 catalog_manager.cc:4830] TS af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377): tablet a959a6c207af45009ff7807ea6f20ee2 (table rebalance_test_table_2 [id=0ddf2a368d7344a8a2eeeaf44782ed5c]) successfully deleted
I20240503 14:13:55.325570 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:55.341276 16960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:55.349967 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:55.373497 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:55.375882 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:55.377723 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:55.395324 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:55.420192 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:55.641726 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:55.651655 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:55.655305 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:55.655364 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:55.658613 16960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:55.659665 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:55.699697 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:55.701361 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:56.093128 13254 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:56.103189 14485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:56.106122 12863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:56.122344 12992 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:56.140288 16960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:56.171362 13122 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:56.176765 13514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:56.197019 13382 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
587e2eb33b9846ca8ab894d34f10c964 | 127.10.145.62:35023 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.145.20:46469 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
21d8b64a50ba4c919cb9e711d18fb7de | 127.10.145.7:44891 | HEALTHY | <none> | 0 | 0
2cd2acf7d8274b5a95482dffd0823a08 | 127.10.145.4:42597 | HEALTHY | <none> | 3 | 0
3678714622584fa4883e4bf07fcabd75 | 127.10.145.2:36441 | HEALTHY | <none> | 5 | 0
6a7786296dae40d4aa2c86869ddfb919 | 127.10.145.8:34135 | HEALTHY | <none> | 0 | 0
72b07e9b53b9471989185277045db8f7 | 127.10.145.1:38531 | HEALTHY | <none> | 2 | 0
7b75d8c9e35a416ba24798fde8b9def0 | 127.10.145.3:41101 | HEALTHY | <none> | 2 | 0
af88d71da3a54f779cf4f651f4393bb6 | 127.10.145.6:36377 | HEALTHY | <none> | 2 | 0
e148d8a3757340ab9288ea98cabb5ad3 | 127.10.145.5:38005 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.10.145.1 | experimental | 127.10.145.1:38531
local_ip_for_outbound_sockets | 127.10.145.2 | experimental | 127.10.145.2:36441
local_ip_for_outbound_sockets | 127.10.145.3 | experimental | 127.10.145.3:41101
local_ip_for_outbound_sockets | 127.10.145.4 | experimental | 127.10.145.4:42597
local_ip_for_outbound_sockets | 127.10.145.5 | experimental | 127.10.145.5:38005
local_ip_for_outbound_sockets | 127.10.145.6 | experimental | 127.10.145.6:36377
local_ip_for_outbound_sockets | 127.10.145.7 | experimental | 127.10.145.7:44891
local_ip_for_outbound_sockets | 127.10.145.8 | experimental | 127.10.145.8:34135
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.145.1:38531
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.145.2:36441
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.145.3:41101
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.10.145.4:42597
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.10.145.5:38005
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.10.145.6:36377
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.10.145.7:44891
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.10.145.8:34135
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.145.20:46469 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:13:56.265771 10820 log_verifier.cc:126] Checking tablet 1bfebd34071946398632ab7597622a41
I20240503 14:13:56.713701 10820 log_verifier.cc:177] Verified matching terms for 468 ops in tablet 1bfebd34071946398632ab7597622a41
I20240503 14:13:56.714934 10820 log_verifier.cc:126] Checking tablet 3117228f0f7c431981bf0173b3b3344f
I20240503 14:13:57.188992 10820 log_verifier.cc:177] Verified matching terms for 576 ops in tablet 3117228f0f7c431981bf0173b3b3344f
I20240503 14:13:57.190368 10820 log_verifier.cc:126] Checking tablet 32029fdaf54f4bccac238b11019ec913
I20240503 14:13:57.816610 10820 log_verifier.cc:177] Verified matching terms for 738 ops in tablet 32029fdaf54f4bccac238b11019ec913
I20240503 14:13:57.818631 10820 log_verifier.cc:126] Checking tablet 37542e9a30324860bb575ae02b3c92b9
I20240503 14:13:58.255270 10820 log_verifier.cc:177] Verified matching terms for 476 ops in tablet 37542e9a30324860bb575ae02b3c92b9
I20240503 14:13:58.256539 10820 log_verifier.cc:126] Checking tablet 3d17c42814e24b70b25c24ea15a336f3
I20240503 14:13:58.737179 10820 log_verifier.cc:177] Verified matching terms for 506 ops in tablet 3d17c42814e24b70b25c24ea15a336f3
I20240503 14:13:58.738461 10820 log_verifier.cc:126] Checking tablet 407844cff18c4421ab36e61e58ab5ee9
I20240503 14:13:59.329530 10820 log_verifier.cc:177] Verified matching terms for 719 ops in tablet 407844cff18c4421ab36e61e58ab5ee9
I20240503 14:13:59.332278 10820 log_verifier.cc:126] Checking tablet 5e130a7b478f4f91af11bef82ebf7236
I20240503 14:13:59.844485 10820 log_verifier.cc:177] Verified matching terms for 636 ops in tablet 5e130a7b478f4f91af11bef82ebf7236
I20240503 14:13:59.846050 10820 log_verifier.cc:126] Checking tablet 64d1257f67b54fb3a4b36f0f2b9f58c3
I20240503 14:14:00.244486 10820 log_verifier.cc:177] Verified matching terms for 441 ops in tablet 64d1257f67b54fb3a4b36f0f2b9f58c3
I20240503 14:14:00.245632 10820 log_verifier.cc:126] Checking tablet 71289cddae864085beacce5aa1fb0c4e
I20240503 14:14:00.794720 10820 log_verifier.cc:177] Verified matching terms for 626 ops in tablet 71289cddae864085beacce5aa1fb0c4e
I20240503 14:14:00.796346 10820 log_verifier.cc:126] Checking tablet 7778260935e54e7d90e6c8d929b2c564
I20240503 14:14:01.152040 10820 log_verifier.cc:177] Verified matching terms for 398 ops in tablet 7778260935e54e7d90e6c8d929b2c564
I20240503 14:14:01.153112 10820 log_verifier.cc:126] Checking tablet 99d215a7d95a41c6a14e7165cf0fe884
I20240503 14:14:01.506798 10820 log_verifier.cc:177] Verified matching terms for 377 ops in tablet 99d215a7d95a41c6a14e7165cf0fe884
I20240503 14:14:01.507679 10820 log_verifier.cc:126] Checking tablet a324330d54ff4a01982e13d0cc191296
I20240503 14:14:01.942571 10820 log_verifier.cc:177] Verified matching terms for 493 ops in tablet a324330d54ff4a01982e13d0cc191296
I20240503 14:14:01.943732 10820 log_verifier.cc:126] Checking tablet a959a6c207af45009ff7807ea6f20ee2
I20240503 14:14:02.550663 10820 log_verifier.cc:177] Verified matching terms for 737 ops in tablet a959a6c207af45009ff7807ea6f20ee2
I20240503 14:14:02.552462 10820 log_verifier.cc:126] Checking tablet c43a1cb5209c4a69a95877c842460adc
I20240503 14:14:02.937642 10820 log_verifier.cc:177] Verified matching terms for 445 ops in tablet c43a1cb5209c4a69a95877c842460adc
I20240503 14:14:02.938962 10820 log_verifier.cc:126] Checking tablet e82229abb5be4ed1abc3fa200671416e
I20240503 14:14:02.969859 10820 log_verifier.cc:177] Verified matching terms for 4 ops in tablet e82229abb5be4ed1abc3fa200671416e
I20240503 14:14:02.969974 10820 log_verifier.cc:126] Checking tablet f99a260ae30a4efda1126bf96b1ac02f
I20240503 14:14:03.321403 10820 log_verifier.cc:177] Verified matching terms for 386 ops in tablet f99a260ae30a4efda1126bf96b1ac02f
I20240503 14:14:03.331324 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 12801
I20240503 14:14:03.344184 12923 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:03.646664 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 12801
W20240503 14:14:03.699093 13469 connection.cc:535] server connection from 127.10.145.1:41635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:03.699317 13467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.145.1:38531 (error 108)
I20240503 14:14:03.699591 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 12932
W20240503 14:14:03.699822 12947 connection.cc:535] client connection to 127.10.145.1:38531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:03.699960 12947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 3 similar messages]
W20240503 14:14:03.711272 13467 consensus_peers.cc:485] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 -> Peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Couldn't send request to peer 72b07e9b53b9471989185277045db8f7. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:03.713616 12947 consensus_peers.cc:485] T 1bfebd34071946398632ab7597622a41 P 3678714622584fa4883e4bf07fcabd75 -> Peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Couldn't send request to peer 72b07e9b53b9471989185277045db8f7. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:03.717844 13053 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:14:03.728159 13207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111)
W20240503 14:14:03.742612 13207 consensus_peers.cc:485] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 -> Peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Couldn't send request to peer 72b07e9b53b9471989185277045db8f7. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:03.776410 13077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111)
W20240503 14:14:03.778127 13077 consensus_peers.cc:485] T 5e130a7b478f4f91af11bef82ebf7236 P 7b75d8c9e35a416ba24798fde8b9def0 -> Peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Couldn't send request to peer 72b07e9b53b9471989185277045db8f7. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:03.922219 13337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111)
W20240503 14:14:03.924023 13337 consensus_peers.cc:485] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 -> Peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Couldn't send request to peer 72b07e9b53b9471989185277045db8f7. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:03.968075 13077 consensus_peers.cc:485] T 3d17c42814e24b70b25c24ea15a336f3 P 7b75d8c9e35a416ba24798fde8b9def0 -> Peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Couldn't send request to peer 72b07e9b53b9471989185277045db8f7. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:03.988548 13337 consensus_peers.cc:485] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 -> Peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Couldn't send request to peer 72b07e9b53b9471989185277045db8f7. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:04.032387 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 12932
W20240503 14:14:04.079181 14438 connection.cc:535] server connection from 127.10.145.2:33607 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.079437 13467 connection.cc:535] client connection to 127.10.145.2:36441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.079916 13078 connection.cc:535] server connection from 127.10.145.2:55399 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:04.079964 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 13062
W20240503 14:14:04.080186 13207 connection.cc:535] client connection to 127.10.145.2:36441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.080971 13467 consensus_peers.cc:485] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 -> Peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Couldn't send request to peer 3678714622584fa4883e4bf07fcabd75. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:04.081629 12728 connection.cc:535] server connection from 127.10.145.2:44169 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.081629 13207 consensus_peers.cc:485] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 -> Peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Couldn't send request to peer 3678714622584fa4883e4bf07fcabd75. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:04.081763 13207 consensus_peers.cc:485] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 -> Peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Couldn't send request to peer 3678714622584fa4883e4bf07fcabd75. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:04.087491 13183 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:14:04.147464 13467 consensus_peers.cc:485] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 -> Peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Couldn't send request to peer 3678714622584fa4883e4bf07fcabd75. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:04.380316 18047 raft_consensus.cc:482] T 407844cff18c4421ab36e61e58ab5ee9 P 7b75d8c9e35a416ba24798fde8b9def0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72b07e9b53b9471989185277045db8f7)
I20240503 14:14:04.380519 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 13062
I20240503 14:14:04.427016 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 13192
W20240503 14:14:04.427516 13339 consensus_peers.cc:485] T 7778260935e54e7d90e6c8d929b2c564 P e148d8a3757340ab9288ea98cabb5ad3 -> Peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Couldn't send request to peer 7b75d8c9e35a416ba24798fde8b9def0. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:04.427641 16916 connection.cc:535] server connection from 127.10.145.3:56765 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.427739 12726 connection.cc:535] server connection from 127.10.145.3:36755 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.428012 13209 consensus_peers.cc:485] T 3117228f0f7c431981bf0173b3b3344f P 2cd2acf7d8274b5a95482dffd0823a08 -> Peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Couldn't send request to peer 7b75d8c9e35a416ba24798fde8b9def0. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:04.428159 13209 consensus_peers.cc:485] T c43a1cb5209c4a69a95877c842460adc P 2cd2acf7d8274b5a95482dffd0823a08 -> Peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Couldn't send request to peer 7b75d8c9e35a416ba24798fde8b9def0. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:04.433938 13314 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:04.568594 18034 raft_consensus.cc:482] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72b07e9b53b9471989185277045db8f7)
I20240503 14:14:04.568817 18034 raft_consensus.cc:504] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } }
I20240503 14:14:04.569877 18034 leader_election.cc:288] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101)
I20240503 14:14:04.570627 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407844cff18c4421ab36e61e58ab5ee9" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 719 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
W20240503 14:14:04.571974 13469 leader_election.cc:334] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111)
W20240503 14:14:04.572201 13467 leader_election.cc:334] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
I20240503 14:14:04.572295 13467 leader_election.cc:302] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: af88d71da3a54f779cf4f651f4393bb6; no voters: 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0, e148d8a3757340ab9288ea98cabb5ad3
I20240503 14:14:04.572646 18034 raft_consensus.cc:2726] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:04.573259 13467 leader_election.cc:334] T 407844cff18c4421ab36e61e58ab5ee9 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111)
W20240503 14:14:04.742566 13209 consensus_peers.cc:485] T 32029fdaf54f4bccac238b11019ec913 P 2cd2acf7d8274b5a95482dffd0823a08 -> Peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Couldn't send request to peer 7b75d8c9e35a416ba24798fde8b9def0. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:04.742955 18044 raft_consensus.cc:482] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72b07e9b53b9471989185277045db8f7)
I20240503 14:14:04.743134 18044 raft_consensus.cc:504] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 2cd2acf7d8274b5a95482dffd0823a08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:04.743350 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 13192
W20240503 14:14:04.791267 14436 connection.cc:535] server connection from 127.10.145.4:38011 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.791440 13468 connection.cc:535] client connection to 127.10.145.4:42597 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.791695 13338 connection.cc:535] client connection to 127.10.145.4:42597 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:04.792084 12728 connection.cc:535] server connection from 127.10.145.4:43541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:04.792619 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 13322
I20240503 14:14:04.792744 18056 raft_consensus.cc:482] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
W20240503 14:14:04.792798 13338 consensus_peers.cc:485] T a959a6c207af45009ff7807ea6f20ee2 P e148d8a3757340ab9288ea98cabb5ad3 -> Peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): Couldn't send request to peer 2cd2acf7d8274b5a95482dffd0823a08. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.4:42597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:04.793059 13468 consensus_peers.cc:485] T f99a260ae30a4efda1126bf96b1ac02f P af88d71da3a54f779cf4f651f4393bb6 -> Peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): Couldn't send request to peer 2cd2acf7d8274b5a95482dffd0823a08. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.4:42597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:04.793174 13339 connection.cc:535] server connection from 127.10.145.4:41931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:04.793048 18056 raft_consensus.cc:504] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:04.795019 18056 leader_election.cc:288] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135)
I20240503 14:14:04.800701 13443 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:04.803350 13530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:14:04.805202 18063 raft_consensus.cc:482] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72b07e9b53b9471989185277045db8f7)
I20240503 14:14:04.805616 18063 raft_consensus.cc:504] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:04.808290 18063 leader_election.cc:288] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:14:04.808756 13530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "6a7786296dae40d4aa2c86869ddfb919" candidate_term: 2 candidate_status { last_received { term: 1 index: 441 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
W20240503 14:14:04.809664 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111)
W20240503 14:14:04.818441 14436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
W20240503 14:14:04.820191 16914 leader_election.cc:334] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): Network error: Client connection negotiation failed: client connection to 127.10.145.4:42597: connect: Connection refused (error 111)
W20240503 14:14:04.822103 16913 leader_election.cc:334] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111)
I20240503 14:14:04.822276 16913 leader_election.cc:302] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 6a7786296dae40d4aa2c86869ddfb919; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, af88d71da3a54f779cf4f651f4393bb6
I20240503 14:14:04.822867 18063 raft_consensus.cc:2726] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:04.824813 14436 leader_election.cc:334] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
I20240503 14:14:04.829649 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6a7786296dae40d4aa2c86869ddfb919" is_pre_election: true
I20240503 14:14:04.830466 14437 leader_election.cc:302] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 21d8b64a50ba4c919cb9e711d18fb7de; no voters: 3678714622584fa4883e4bf07fcabd75, 6a7786296dae40d4aa2c86869ddfb919, af88d71da3a54f779cf4f651f4393bb6
I20240503 14:14:04.830897 18056 raft_consensus.cc:2726] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:04.903738 18063 raft_consensus.cc:482] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:14:04.903970 18063 raft_consensus.cc:504] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:04.905015 18063 leader_election.cc:288] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
I20240503 14:14:04.905727 13530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "6a7786296dae40d4aa2c86869ddfb919" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
W20240503 14:14:04.906782 16913 leader_election.cc:334] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
I20240503 14:14:04.911949 14506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "6a7786296dae40d4aa2c86869ddfb919" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" is_pre_election: true
I20240503 14:14:04.912264 14506 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7786296dae40d4aa2c86869ddfb919 in term 1.
I20240503 14:14:04.959781 18056 raft_consensus.cc:482] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:14:04.960007 18056 raft_consensus.cc:504] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:14:04.961380 18056 leader_election.cc:288] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597)
I20240503 14:14:04.961897 13530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a324330d54ff4a01982e13d0cc191296" candidate_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 493 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
W20240503 14:14:04.962733 14438 leader_election.cc:334] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111)
W20240503 14:14:04.963209 14436 leader_election.cc:334] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
I20240503 14:14:04.963325 14436 leader_election.cc:302] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 21d8b64a50ba4c919cb9e711d18fb7de; no voters: 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0, af88d71da3a54f779cf4f651f4393bb6
I20240503 14:14:04.963593 18056 raft_consensus.cc:2726] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:04.964030 14437 leader_election.cc:334] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): Network error: Client connection negotiation failed: client connection to 127.10.145.4:42597: connect: Connection refused (error 111)
I20240503 14:14:04.967643 18035 raft_consensus.cc:482] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:14:04.967849 18035 raft_consensus.cc:504] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:04.968838 18035 leader_election.cc:288] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891), 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135)
I20240503 14:14:04.969466 14506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" is_pre_election: true
I20240503 14:14:04.969599 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6a7786296dae40d4aa2c86869ddfb919" is_pre_election: true
I20240503 14:14:04.969704 14506 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
I20240503 14:14:04.969857 16977 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
W20240503 14:14:04.970222 13467 leader_election.cc:334] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
I20240503 14:14:04.970484 13468 leader_election.cc:302] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 21d8b64a50ba4c919cb9e711d18fb7de, 6a7786296dae40d4aa2c86869ddfb919, af88d71da3a54f779cf4f651f4393bb6; no voters: 3678714622584fa4883e4bf07fcabd75
I20240503 14:14:04.970754 18035 raft_consensus.cc:2781] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:14:04.970840 18035 raft_consensus.cc:482] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:14:04.970932 18035 raft_consensus.cc:3037] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:04.972817 18035 raft_consensus.cc:504] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:04.974117 14506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21d8b64a50ba4c919cb9e711d18fb7de"
I20240503 14:14:04.974184 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82229abb5be4ed1abc3fa200671416e" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6a7786296dae40d4aa2c86869ddfb919"
I20240503 14:14:04.974330 14506 raft_consensus.cc:3037] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:04.974396 16977 raft_consensus.cc:3037] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:14:04.974887 13467 leader_election.cc:334] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
I20240503 14:14:04.973661 18035 leader_election.cc:288] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 election: Requested vote from peers 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891), 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135)
I20240503 14:14:04.977340 16977 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 2.
I20240503 14:14:04.977329 14506 raft_consensus.cc:2454] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 2.
I20240503 14:14:04.977875 13468 leader_election.cc:302] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 21d8b64a50ba4c919cb9e711d18fb7de, 6a7786296dae40d4aa2c86869ddfb919, af88d71da3a54f779cf4f651f4393bb6; no voters: 3678714622584fa4883e4bf07fcabd75
I20240503 14:14:04.978185 18035 raft_consensus.cc:2781] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:04.978348 18035 raft_consensus.cc:686] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [term 2 LEADER]: Becoming Leader. State: Replica: af88d71da3a54f779cf4f651f4393bb6, State: Running, Role: LEADER
I20240503 14:14:04.978669 18035 consensus_queue.cc:237] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:04.981999 12733 catalog_manager.cc:5484] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 reported cstate change: term changed from 1 to 2, leader changed from 3678714622584fa4883e4bf07fcabd75 (127.10.145.2) to af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6). New cstate: current_term: 2 leader_uuid: "af88d71da3a54f779cf4f651f4393bb6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:05.009801 18035 raft_consensus.cc:482] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:14:05.010021 18035 raft_consensus.cc:504] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:14:05.010859 18035 leader_election.cc:288] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
I20240503 14:14:05.011920 14506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a324330d54ff4a01982e13d0cc191296" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 493 } } ignore_live_leader: false dest_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" is_pre_election: true
I20240503 14:14:05.012198 14506 raft_consensus.cc:2454] T a324330d54ff4a01982e13d0cc191296 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
W20240503 14:14:05.012851 13468 leader_election.cc:334] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): Network error: Client connection negotiation failed: client connection to 127.10.145.4:42597: connect: Connection refused (error 111)
W20240503 14:14:05.013041 13469 leader_election.cc:334] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111)
W20240503 14:14:05.013175 13467 leader_election.cc:334] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
I20240503 14:14:05.013334 13467 leader_election.cc:302] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 21d8b64a50ba4c919cb9e711d18fb7de, af88d71da3a54f779cf4f651f4393bb6; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0
I20240503 14:14:05.013609 18035 raft_consensus.cc:2726] T a324330d54ff4a01982e13d0cc191296 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:05.057260 18035 raft_consensus.cc:482] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:14:05.057543 18035 raft_consensus.cc:504] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } }
I20240503 14:14:05.058419 18035 leader_election.cc:288] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441)
W20240503 14:14:05.059804 13469 leader_election.cc:334] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111)
W20240503 14:14:05.060156 13467 leader_election.cc:334] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111)
W20240503 14:14:05.060335 13468 leader_election.cc:334] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): Network error: Client connection negotiation failed: client connection to 127.10.145.4:42597: connect: Connection refused (error 111)
I20240503 14:14:05.060479 13468 leader_election.cc:302] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: af88d71da3a54f779cf4f651f4393bb6; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, 7b75d8c9e35a416ba24798fde8b9def0
I20240503 14:14:05.060739 18035 raft_consensus.cc:2726] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:05.060875 13467 leader_election.cc:334] T 1bfebd34071946398632ab7597622a41 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
I20240503 14:14:05.112103 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "6a7786296dae40d4aa2c86869ddfb919" candidate_term: 2 candidate_status { last_received { term: 1 index: 441 } } ignore_live_leader: false dest_uuid: "e148d8a3757340ab9288ea98cabb5ad3" is_pre_election: true
I20240503 14:14:05.113399 18049 raft_consensus.cc:482] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:14:05.114180 18049 raft_consensus.cc:504] T 37542e9a30324860bb575ae02b3c92b9 P e148d8a3757340ab9288ea98cabb5ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:14:05.114823 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 13322
I20240503 14:14:05.149598 18035 raft_consensus.cc:482] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72b07e9b53b9471989185277045db8f7)
I20240503 14:14:05.149852 18035 raft_consensus.cc:504] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:05.150808 18035 leader_election.cc:288] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135)
I20240503 14:14:05.151787 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d1257f67b54fb3a4b36f0f2b9f58c3" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 441 } } ignore_live_leader: false dest_uuid: "6a7786296dae40d4aa2c86869ddfb919" is_pre_election: true
I20240503 14:14:05.152091 16977 raft_consensus.cc:2454] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af88d71da3a54f779cf4f651f4393bb6 in term 1.
W20240503 14:14:05.152209 13467 leader_election.cc:334] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111)
W20240503 14:14:05.152930 13468 leader_election.cc:334] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): Network error: Client connection negotiation failed: client connection to 127.10.145.4:42597: connect: Connection refused (error 111)
W20240503 14:14:05.166924 16914 connection.cc:535] client connection to 127.10.145.5:38005 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:05.167101 13468 connection.cc:535] client connection to 127.10.145.5:38005 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:05.168303 18059 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:14:04.795641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.145.5:38005 (local address 127.10.145.7:40085)
0503 14:14:04.801153 (+ 5512us) negotiation.cc:107] Waiting for socket to connect
0503 14:14:04.801173 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0503 14:14:04.801306 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:14:04.803496 (+ 2190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 14:14:04.803501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 14:14:04.803535 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 14:14:04.803886 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 14:14:04.803900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 14:14:05.167156 (+363256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: BlockingRecv error: recv got EOF from 127.10.145.5:38005 (error 108)
Metrics: {"client-negotiator.queue_time_us":5412,"thread_start_us":22150,"threads_started":1}
I20240503 14:14:05.169108 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 13452
W20240503 14:14:05.169715 14437 leader_election.cc:334] T e82229abb5be4ed1abc3fa200671416e P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: connect: Connection refused (error 111)
W20240503 14:14:05.169924 18076 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:14:05.167843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.145.5:38005 (local address 127.10.145.6:52735)
0503 14:14:05.168012 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0503 14:14:05.168032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0503 14:14:05.168170 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:14:05.169126 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":52}
W20240503 14:14:05.169950 18066 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:14:05.168245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.145.5:38005 (local address 127.10.145.8:57511)
0503 14:14:05.168399 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0503 14:14:05.168415 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0503 14:14:05.168527 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:14:05.169056 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":43}
W20240503 14:14:05.170150 16914 leader_election.cc:334] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:05.170233 13468 consensus_peers.cc:485] T 71289cddae864085beacce5aa1fb0c4e P af88d71da3a54f779cf4f651f4393bb6 -> Peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Couldn't send request to peer e148d8a3757340ab9288ea98cabb5ad3. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:05.170347 13468 leader_election.cc:334] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:05.170648 13468 leader_election.cc:334] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:05.170673 16914 leader_election.cc:334] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:05.170763 13468 leader_election.cc:302] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 6a7786296dae40d4aa2c86869ddfb919, af88d71da3a54f779cf4f651f4393bb6; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 72b07e9b53b9471989185277045db8f7, e148d8a3757340ab9288ea98cabb5ad3
I20240503 14:14:05.170794 16914 leader_election.cc:302] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 21d8b64a50ba4c919cb9e711d18fb7de, 6a7786296dae40d4aa2c86869ddfb919; no voters: 3678714622584fa4883e4bf07fcabd75, af88d71da3a54f779cf4f651f4393bb6, e148d8a3757340ab9288ea98cabb5ad3
I20240503 14:14:05.171119 18035 raft_consensus.cc:2726] T 64d1257f67b54fb3a4b36f0f2b9f58c3 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:05.171218 13468 leader_election.cc:334] T e82229abb5be4ed1abc3fa200671416e P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:05.171705 18063 raft_consensus.cc:2726] T e82229abb5be4ed1abc3fa200671416e P 6a7786296dae40d4aa2c86869ddfb919 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:05.177538 13574 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:05.215739 18056 raft_consensus.cc:482] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:14:05.215958 18056 raft_consensus.cc:504] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:14:05.216981 18056 leader_election.cc:288] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), af88d71da3a54f779cf4f651f4393bb6 (127.10.145.6:36377)
W20240503 14:14:05.218509 14436 leader_election.cc:334] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
W20240503 14:14:05.218662 14437 leader_election.cc:334] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): Network error: Client connection negotiation failed: client connection to 127.10.145.4:42597: connect: Connection refused (error 111)
W20240503 14:14:05.218889 14438 leader_election.cc:334] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111)
I20240503 14:14:05.219033 14438 leader_election.cc:302] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 21d8b64a50ba4c919cb9e711d18fb7de; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0
I20240503 14:14:05.219280 18056 raft_consensus.cc:2726] T 99d215a7d95a41c6a14e7165cf0fe884 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:05.447208 18056 raft_consensus.cc:482] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b75d8c9e35a416ba24798fde8b9def0)
I20240503 14:14:05.447454 18056 raft_consensus.cc:504] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:05.448513 18056 leader_election.cc:288] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 6a7786296dae40d4aa2c86869ddfb919 (127.10.145.8:34135)
I20240503 14:14:05.449538 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d17c42814e24b70b25c24ea15a336f3" candidate_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 506 } } ignore_live_leader: false dest_uuid: "6a7786296dae40d4aa2c86869ddfb919" is_pre_election: true
W20240503 14:14:05.450516 14437 leader_election.cc:334] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: connect: Connection refused (error 111)
W20240503 14:14:05.450707 14436 leader_election.cc:334] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111)
I20240503 14:14:05.450834 14436 leader_election.cc:302] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 21d8b64a50ba4c919cb9e711d18fb7de; no voters: 6a7786296dae40d4aa2c86869ddfb919, 72b07e9b53b9471989185277045db8f7, e148d8a3757340ab9288ea98cabb5ad3
W20240503 14:14:05.463996 14438 leader_election.cc:334] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111)
I20240503 14:14:05.464462 13530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d215a7d95a41c6a14e7165cf0fe884" candidate_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 377 } } ignore_live_leader: false dest_uuid: "af88d71da3a54f779cf4f651f4393bb6" is_pre_election: true
I20240503 14:14:05.464694 18035 raft_consensus.cc:482] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:14:05.465411 18056 raft_consensus.cc:2726] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:05.465646 18035 raft_consensus.cc:504] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:14:05.466554 18035 leader_election.cc:288] T 37542e9a30324860bb575ae02b3c92b9 P af88d71da3a54f779cf4f651f4393bb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
I20240503 14:14:05.466935 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 13452
I20240503 14:14:05.468305 14506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37542e9a30324860bb575ae02b3c92b9" candidate_uuid: "af88d71da3a54f779cf4f651f4393bb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 476 } } ignore_live_leader: false dest_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" is_pre_election: true
I20240503 14:14:05.511482 18063 raft_consensus.cc:482] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b75d8c9e35a416ba24798fde8b9def0)
I20240503 14:14:05.511731 18063 raft_consensus.cc:504] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 506 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:05.513020 18063 leader_election.cc:288] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
I20240503 14:14:05.513958 14506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d17c42814e24b70b25c24ea15a336f3" candidate_uuid: "6a7786296dae40d4aa2c86869ddfb919" candidate_term: 2 candidate_status { last_received { term: 1 index: 506 } } ignore_live_leader: false dest_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" is_pre_election: true
I20240503 14:14:05.514240 14506 raft_consensus.cc:2454] T 3d17c42814e24b70b25c24ea15a336f3 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7786296dae40d4aa2c86869ddfb919 in term 1.
W20240503 14:14:05.515141 16914 leader_election.cc:334] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: connect: Connection refused (error 111)
W20240503 14:14:05.515410 16915 leader_election.cc:334] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111)
W20240503 14:14:05.515905 16913 leader_election.cc:334] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 72b07e9b53b9471989185277045db8f7 (127.10.145.1:38531): Network error: Client connection negotiation failed: client connection to 127.10.145.1:38531: connect: Connection refused (error 111)
I20240503 14:14:05.516026 16913 leader_election.cc:302] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 21d8b64a50ba4c919cb9e711d18fb7de, 6a7786296dae40d4aa2c86869ddfb919; no voters: 72b07e9b53b9471989185277045db8f7, 7b75d8c9e35a416ba24798fde8b9def0, e148d8a3757340ab9288ea98cabb5ad3
W20240503 14:14:05.516072 14436 connection.cc:535] server connection from 127.10.145.6:36979 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:05.516484 16913 connection.cc:535] server connection from 127.10.145.6:37475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:05.516646 18063 raft_consensus.cc:2726] T 3d17c42814e24b70b25c24ea15a336f3 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:05.516857 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 14418
I20240503 14:14:05.522490 14546 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:05.549777 18063 raft_consensus.cc:482] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2cd2acf7d8274b5a95482dffd0823a08)
I20240503 14:14:05.550074 18063 raft_consensus.cc:504] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 738 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:05.551368 18063 leader_election.cc:288] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
W20240503 14:14:05.553270 16914 leader_election.cc:334] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): Network error: Client connection negotiation failed: client connection to 127.10.145.4:42597: connect: Connection refused (error 111)
W20240503 14:14:05.553536 16915 leader_election.cc:334] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111)
W20240503 14:14:05.553755 16913 leader_election.cc:334] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3678714622584fa4883e4bf07fcabd75 (127.10.145.2:36441): Network error: Client connection negotiation failed: client connection to 127.10.145.2:36441: connect: Connection refused (error 111)
I20240503 14:14:05.553931 16913 leader_election.cc:302] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 6a7786296dae40d4aa2c86869ddfb919; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 3678714622584fa4883e4bf07fcabd75, 7b75d8c9e35a416ba24798fde8b9def0
I20240503 14:14:05.554225 18063 raft_consensus.cc:2726] T 32029fdaf54f4bccac238b11019ec913 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:05.779754 14506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32029fdaf54f4bccac238b11019ec913" candidate_uuid: "6a7786296dae40d4aa2c86869ddfb919" candidate_term: 2 candidate_status { last_received { term: 1 index: 738 } } ignore_live_leader: false dest_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" is_pre_election: true
I20240503 14:14:05.780066 18056 raft_consensus.cc:482] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3678714622584fa4883e4bf07fcabd75)
I20240503 14:14:05.780767 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 14418
I20240503 14:14:05.780830 18056 raft_consensus.cc:504] T 37542e9a30324860bb575ae02b3c92b9 P 21d8b64a50ba4c919cb9e711d18fb7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "3678714622584fa4883e4bf07fcabd75" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 36441 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } }
I20240503 14:14:05.826716 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 16896
I20240503 14:14:05.827842 18063 raft_consensus.cc:482] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2cd2acf7d8274b5a95482dffd0823a08)
I20240503 14:14:05.828055 18063 raft_consensus.cc:504] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 445 OBSOLETE_local: false peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "e148d8a3757340ab9288ea98cabb5ad3" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 38005 } } peers { permanent_uuid: "21d8b64a50ba4c919cb9e711d18fb7de" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:05.828995 18063 leader_election.cc:288] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101), 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597), e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005), 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891)
W20240503 14:14:05.830332 16915 leader_election.cc:334] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b75d8c9e35a416ba24798fde8b9def0 (127.10.145.3:41101): Network error: Client connection negotiation failed: client connection to 127.10.145.3:41101: connect: Connection refused (error 111)
W20240503 14:14:05.831800 16914 leader_election.cc:334] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cd2acf7d8274b5a95482dffd0823a08 (127.10.145.4:42597): Network error: Client connection negotiation failed: client connection to 127.10.145.4:42597: connect: Connection refused (error 111)
W20240503 14:14:05.832024 16914 leader_election.cc:334] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e148d8a3757340ab9288ea98cabb5ad3 (127.10.145.5:38005): Network error: Client connection negotiation failed: client connection to 127.10.145.5:38005: connect: Connection refused (error 111)
I20240503 14:14:05.832140 16914 leader_election.cc:302] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 6a7786296dae40d4aa2c86869ddfb919; no voters: 2cd2acf7d8274b5a95482dffd0823a08, 7b75d8c9e35a416ba24798fde8b9def0, e148d8a3757340ab9288ea98cabb5ad3
I20240503 14:14:05.832439 18063 raft_consensus.cc:2726] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:05.832465 16914 leader_election.cc:334] T c43a1cb5209c4a69a95877c842460adc P 6a7786296dae40d4aa2c86869ddfb919 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 21d8b64a50ba4c919cb9e711d18fb7de (127.10.145.7:44891): Network error: Client connection negotiation failed: client connection to 127.10.145.7:44891: connect: Connection refused (error 111)
I20240503 14:14:05.834115 17019 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:06.018601 18063 raft_consensus.cc:482] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b75d8c9e35a416ba24798fde8b9def0)
I20240503 14:14:06.018896 18063 raft_consensus.cc:504] T 5e130a7b478f4f91af11bef82ebf7236 P 6a7786296dae40d4aa2c86869ddfb919 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 636 OBSOLETE_local: false peers { permanent_uuid: "2cd2acf7d8274b5a95482dffd0823a08" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 42597 } } peers { permanent_uuid: "72b07e9b53b9471989185277045db8f7" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 38531 } } peers { permanent_uuid: "7b75d8c9e35a416ba24798fde8b9def0" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 41101 } } peers { permanent_uuid: "af88d71da3a54f779cf4f651f4393bb6" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36377 } } peers { permanent_uuid: "6a7786296dae40d4aa2c86869ddfb919" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 34135 } attrs { promote: false } }
I20240503 14:14:06.019162 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 16896
I20240503 14:14:06.045228 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 12711
I20240503 14:14:06.046356 12772 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:06.239528 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 12711
2024-05-03T14:14:06Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/7 (113552 ms)
[----------] 1 test from RebalanceParamTest (113552 ms total)
[----------] 1 test from TserverAddedDuringRebalancingTest
[ RUN ] TserverAddedDuringRebalancingTest.TserverStarts/1
I20240503 14:14:06.378358 10820 test_util.cc:268] Using random seed: 1428286796
I20240503 14:14:06.380331 10820 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:14:06.380411 10820 ts_itest-base.cc:113] --------------
I20240503 14:14:06.380450 10820 ts_itest-base.cc:114] 8 tablet servers
I20240503 14:14:06.380522 10820 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:14:06.380599 10820 ts_itest-base.cc:116] --------------
2024-05-03T14:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:14:06Z Disabled control of system clock
I20240503 14:14:06.467504 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:33209
--webserver_interface=127.10.145.62
--webserver_port=0
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.145.62:33209
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 14:14:06.820698 18097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:06.821049 18097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:06.821161 18097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:06.828755 18097 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:06.828908 18097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:14:06.828990 18097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:06.829070 18097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:14:06.829131 18097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:14:06.838546 18097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.10.145.62:33209
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:33209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:06.840029 18097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:06.841955 18097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:06.853364 18103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:06.853591 18102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:06.854099 18105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:06.854557 18097 server_base.cc:1034] running on GCE node
I20240503 14:14:06.855633 18097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:06.857699 18097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:06.858980 18097 hybrid_clock.cc:648] HybridClock initialized: now 1714745646858882 us; error 76 us; skew 500 ppm
I20240503 14:14:06.859403 18097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:06.862068 18097 webserver.cc:460] Webserver started at http://127.10.145.62:42971/ using document root <none> and password file <none>
I20240503 14:14:06.862924 18097 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:06.863068 18097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:06.863432 18097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:06.866437 18097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4127b9518b394f33afa8da3f0c1e0072"
format_stamp: "Formatted at 2024-05-03 14:14:06 on dist-test-slave-v549"
I20240503 14:14:06.867362 18097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4127b9518b394f33afa8da3f0c1e0072"
format_stamp: "Formatted at 2024-05-03 14:14:06 on dist-test-slave-v549"
I20240503 14:14:06.873634 18097 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 14:14:06.877584 18111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:06.879367 18097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:14:06.879595 18097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4127b9518b394f33afa8da3f0c1e0072"
format_stamp: "Formatted at 2024-05-03 14:14:06 on dist-test-slave-v549"
I20240503 14:14:06.879810 18097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:06.895128 18097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:06.896072 18097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:06.896358 18097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:06.913463 18097 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.62:33209
I20240503 14:14:06.913487 18162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.62:33209 every 8 connection(s)
I20240503 14:14:06.914960 18097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:14:06.918848 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 18097
I20240503 14:14:06.919083 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:14:06.919102 18163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:06.932864 18163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072: Bootstrap starting.
I20240503 14:14:06.936268 18163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:06.937315 18163 log.cc:824] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:06.939908 18163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072: No bootstrap required, opened a new log
I20240503 14:14:06.946686 18163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b9518b394f33afa8da3f0c1e0072" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33209 } }
I20240503 14:14:06.947081 18163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:06.947160 18163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127b9518b394f33afa8da3f0c1e0072, State: Initialized, Role: FOLLOWER
I20240503 14:14:06.947765 18163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b9518b394f33afa8da3f0c1e0072" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33209 } }
I20240503 14:14:06.947957 18163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:06.948065 18163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:06.948186 18163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:06.950321 18163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b9518b394f33afa8da3f0c1e0072" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33209 } }
I20240503 14:14:06.950801 18163 leader_election.cc:302] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4127b9518b394f33afa8da3f0c1e0072; no voters:
I20240503 14:14:06.951418 18163 leader_election.cc:288] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:06.951547 18168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:06.952545 18168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [term 1 LEADER]: Becoming Leader. State: Replica: 4127b9518b394f33afa8da3f0c1e0072, State: Running, Role: LEADER
I20240503 14:14:06.953116 18168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b9518b394f33afa8da3f0c1e0072" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33209 } }
I20240503 14:14:06.953609 18163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:14:06.956063 18170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4127b9518b394f33afa8da3f0c1e0072. Latest consensus state: current_term: 1 leader_uuid: "4127b9518b394f33afa8da3f0c1e0072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b9518b394f33afa8da3f0c1e0072" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33209 } } }
I20240503 14:14:06.956480 18170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [sys.catalog]: This master's current role is: LEADER
I20240503 14:14:06.956041 18169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4127b9518b394f33afa8da3f0c1e0072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b9518b394f33afa8da3f0c1e0072" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33209 } } }
I20240503 14:14:06.957123 18169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072 [sys.catalog]: This master's current role is: LEADER
I20240503 14:14:06.958317 18176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:14:06.963795 18176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:14:06.973320 18176 catalog_manager.cc:1347] Generated new cluster ID: 75b1e2f15b55411c8027083f0fab236f
I20240503 14:14:06.973476 18176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:14:06.989984 18176 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:14:06.991261 18176 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:14:07.004173 18176 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4127b9518b394f33afa8da3f0c1e0072: Generated new TSK 0
I20240503 14:14:07.005141 18176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:14:07.018272 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.1:0
--local_ip_for_outbound_sockets=127.10.145.1
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33209
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 14:14:07.385692 18187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:07.386091 18187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:07.386250 18187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:07.394124 18187 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:07.394264 18187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:07.394420 18187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.1
I20240503 14:14:07.403885 18187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.145.1
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.1
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:07.405367 18187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:07.407254 18187 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:07.420004 18192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:07.420197 18193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:07.420701 18195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:07.421370 18187 server_base.cc:1034] running on GCE node
I20240503 14:14:07.422055 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:07.423097 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:07.424301 18187 hybrid_clock.cc:648] HybridClock initialized: now 1714745647424246 us; error 46 us; skew 500 ppm
I20240503 14:14:07.424646 18187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:07.427103 18187 webserver.cc:460] Webserver started at http://127.10.145.1:33057/ using document root <none> and password file <none>
I20240503 14:14:07.427874 18187 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:07.427980 18187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:07.428321 18187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:07.431133 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7ceaaad4bc544c4487859a6671e66838"
format_stamp: "Formatted at 2024-05-03 14:14:07 on dist-test-slave-v549"
I20240503 14:14:07.431931 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7ceaaad4bc544c4487859a6671e66838"
format_stamp: "Formatted at 2024-05-03 14:14:07 on dist-test-slave-v549"
I20240503 14:14:07.437590 18187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20240503 14:14:07.441213 18201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:07.442677 18187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:14:07.442844 18187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7ceaaad4bc544c4487859a6671e66838"
format_stamp: "Formatted at 2024-05-03 14:14:07 on dist-test-slave-v549"
I20240503 14:14:07.443065 18187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:07.470577 18187 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:07.471541 18187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:07.471822 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:07.472851 18187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:07.474589 18187 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:07.474684 18187 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:07.474797 18187 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:07.474874 18187 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:07.503762 18187 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.1:33655
I20240503 14:14:07.503885 18313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.1:33655 every 8 connection(s)
I20240503 14:14:07.505216 18187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:14:07.507159 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 18187
I20240503 14:14:07.507406 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:14:07.511358 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.2:0
--local_ip_for_outbound_sockets=127.10.145.2
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33209
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:07.520820 18314 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:07.521195 18314 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:07.522177 18314 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:07.524525 18128 ts_manager.cc:194] Registered new tserver with Master: 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:07.526157 18128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.1:48539
W20240503 14:14:07.863898 18318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:07.864253 18318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:07.864387 18318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:07.872102 18318 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:07.872268 18318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:07.872439 18318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.2
I20240503 14:14:07.881786 18318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.145.2
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.2
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:07.883221 18318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:07.885121 18318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:07.897691 18324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:07.898130 18326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:07.898429 18323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:07.898821 18318 server_base.cc:1034] running on GCE node
I20240503 14:14:07.899425 18318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:07.900398 18318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:07.901592 18318 hybrid_clock.cc:648] HybridClock initialized: now 1714745647901531 us; error 50 us; skew 500 ppm
I20240503 14:14:07.901902 18318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:07.904196 18318 webserver.cc:460] Webserver started at http://127.10.145.2:43171/ using document root <none> and password file <none>
I20240503 14:14:07.904949 18318 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:07.905052 18318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:07.905377 18318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:07.908073 18318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ea366e6e15a24c52b1ea5a154f942568"
format_stamp: "Formatted at 2024-05-03 14:14:07 on dist-test-slave-v549"
I20240503 14:14:07.908828 18318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ea366e6e15a24c52b1ea5a154f942568"
format_stamp: "Formatted at 2024-05-03 14:14:07 on dist-test-slave-v549"
I20240503 14:14:07.914914 18318 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20240503 14:14:07.918716 18332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:07.920260 18318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:14:07.920424 18318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ea366e6e15a24c52b1ea5a154f942568"
format_stamp: "Formatted at 2024-05-03 14:14:07 on dist-test-slave-v549"
I20240503 14:14:07.920634 18318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:07.950768 18318 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:07.951714 18318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:07.952000 18318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:07.953049 18318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:07.954748 18318 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:07.954846 18318 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:07.954954 18318 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:07.955034 18318 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:07.985507 18318 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.2:43191
I20240503 14:14:07.985626 18444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.2:43191 every 8 connection(s)
I20240503 14:14:07.986919 18318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:14:07.987293 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 18318
I20240503 14:14:07.987479 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:14:07.991518 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.3:0
--local_ip_for_outbound_sockets=127.10.145.3
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33209
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:08.002418 18445 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:08.002780 18445 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:08.003679 18445 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:08.005489 18128 ts_manager.cc:194] Registered new tserver with Master: ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:08.006314 18128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.2:57445
W20240503 14:14:08.358769 18448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:08.359136 18448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:08.359292 18448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:08.367286 18448 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:08.367507 18448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:08.367697 18448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.3
I20240503 14:14:08.377913 18448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.145.3
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.3
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:08.379428 18448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:08.381424 18448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:08.394162 18454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:08.394454 18453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:08.394850 18456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:08.395242 18448 server_base.cc:1034] running on GCE node
I20240503 14:14:08.395925 18448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:08.397087 18448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:08.398319 18448 hybrid_clock.cc:648] HybridClock initialized: now 1714745648398255 us; error 48 us; skew 500 ppm
I20240503 14:14:08.398681 18448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:08.407018 18448 webserver.cc:460] Webserver started at http://127.10.145.3:36571/ using document root <none> and password file <none>
I20240503 14:14:08.408244 18448 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:08.408447 18448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:08.408975 18448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:08.413611 18448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "55339e38e7aa4bb7b7b42aa206108722"
format_stamp: "Formatted at 2024-05-03 14:14:08 on dist-test-slave-v549"
I20240503 14:14:08.414726 18448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "55339e38e7aa4bb7b7b42aa206108722"
format_stamp: "Formatted at 2024-05-03 14:14:08 on dist-test-slave-v549"
I20240503 14:14:08.422912 18448 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20240503 14:14:08.428004 18462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:08.430040 18448 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240503 14:14:08.430231 18448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "55339e38e7aa4bb7b7b42aa206108722"
format_stamp: "Formatted at 2024-05-03 14:14:08 on dist-test-slave-v549"
I20240503 14:14:08.430480 18448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:08.461237 18448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:08.462229 18448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:08.462520 18448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:08.463541 18448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:08.465186 18448 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:08.465305 18448 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:08.465423 18448 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:08.465499 18448 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:08.495157 18448 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.3:38857
I20240503 14:14:08.495216 18574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.3:38857 every 8 connection(s)
I20240503 14:14:08.496577 18448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:14:08.498255 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 18448
I20240503 14:14:08.498446 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:14:08.502112 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.4:0
--local_ip_for_outbound_sockets=127.10.145.4
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33209
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:08.512742 18575 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:08.513135 18575 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:08.514256 18575 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:08.516057 18128 ts_manager.cc:194] Registered new tserver with Master: 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:08.516887 18128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.3:39351
I20240503 14:14:08.530053 18314 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
W20240503 14:14:08.859030 18578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:08.859354 18578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:08.859474 18578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:08.867070 18578 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:08.867208 18578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:08.867336 18578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.4
I20240503 14:14:08.876698 18578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.145.4
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.4
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:08.878127 18578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:08.880120 18578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:08.893180 18584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:08.893368 18583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:08.893833 18586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:08.898581 18578 server_base.cc:1034] running on GCE node
I20240503 14:14:08.899309 18578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:08.900449 18578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:08.901669 18578 hybrid_clock.cc:648] HybridClock initialized: now 1714745648901620 us; error 44 us; skew 500 ppm
I20240503 14:14:08.902047 18578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:08.904479 18578 webserver.cc:460] Webserver started at http://127.10.145.4:42405/ using document root <none> and password file <none>
I20240503 14:14:08.905351 18578 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:08.905483 18578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:08.905814 18578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:08.908663 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1c72210d694344d4b4604140088a6471"
format_stamp: "Formatted at 2024-05-03 14:14:08 on dist-test-slave-v549"
I20240503 14:14:08.909564 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1c72210d694344d4b4604140088a6471"
format_stamp: "Formatted at 2024-05-03 14:14:08 on dist-test-slave-v549"
I20240503 14:14:08.915410 18578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:14:08.919205 18592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:08.920745 18578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:14:08.920923 18578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1c72210d694344d4b4604140088a6471"
format_stamp: "Formatted at 2024-05-03 14:14:08 on dist-test-slave-v549"
I20240503 14:14:08.921145 18578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:08.941761 18578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:08.942719 18578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:08.943017 18578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:08.944059 18578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:08.945770 18578 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:08.945868 18578 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:08.945989 18578 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:08.946064 18578 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:08.977833 18578 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.4:36411
I20240503 14:14:08.977891 18704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.4:36411 every 8 connection(s)
I20240503 14:14:08.979359 18578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:14:08.988713 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 18578
I20240503 14:14:08.988929 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:14:08.993109 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.5:0
--local_ip_for_outbound_sockets=127.10.145.5
--webserver_interface=127.10.145.5
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33209
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:08.993223 18705 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:08.993587 18705 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:08.994482 18705 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:08.996095 18128 ts_manager.cc:194] Registered new tserver with Master: 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:08.996917 18128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.4:39575
I20240503 14:14:09.009640 18445 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
W20240503 14:14:09.355146 18708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:09.355484 18708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:09.355616 18708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:09.363464 18708 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:09.363611 18708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:09.363788 18708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.5
I20240503 14:14:09.373740 18708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.10.145.5
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.5
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:09.375237 18708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:09.377147 18708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:09.391084 18714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:09.391074 18716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:09.391667 18713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:09.392246 18708 server_base.cc:1034] running on GCE node
I20240503 14:14:09.393010 18708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:09.394155 18708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:09.395375 18708 hybrid_clock.cc:648] HybridClock initialized: now 1714745649395321 us; error 52 us; skew 500 ppm
I20240503 14:14:09.395717 18708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:09.398262 18708 webserver.cc:460] Webserver started at http://127.10.145.5:33243/ using document root <none> and password file <none>
I20240503 14:14:09.399099 18708 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:09.399226 18708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:09.399564 18708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:09.402465 18708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "2e6a3f22c7ae444e96f6d35cca602e62"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-v549"
I20240503 14:14:09.403398 18708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "2e6a3f22c7ae444e96f6d35cca602e62"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-v549"
I20240503 14:14:09.409140 18708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:14:09.414074 18722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:09.415753 18708 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:14:09.415954 18708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2e6a3f22c7ae444e96f6d35cca602e62"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-v549"
I20240503 14:14:09.416180 18708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:09.430977 18708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:09.431921 18708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:09.432211 18708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:09.433357 18708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:09.435022 18708 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:09.435120 18708 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:09.435237 18708 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:09.435303 18708 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:09.466421 18708 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.5:40605
I20240503 14:14:09.466504 18834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.5:40605 every 8 connection(s)
I20240503 14:14:09.467824 18708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:14:09.468686 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 18708
I20240503 14:14:09.468928 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:14:09.472662 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.6:0
--local_ip_for_outbound_sockets=127.10.145.6
--webserver_interface=127.10.145.6
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33209
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:09.485095 18835 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:09.485643 18835 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:09.486634 18835 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:09.488359 18128 ts_manager.cc:194] Registered new tserver with Master: 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605)
I20240503 14:14:09.489218 18128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.5:37585
I20240503 14:14:09.520205 18575 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
W20240503 14:14:09.833776 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:09.834128 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:09.834275 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:09.842108 18838 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:09.842268 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:09.842438 18838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.6
I20240503 14:14:09.851907 18838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.10.145.6
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.6
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:09.853430 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:09.855444 18838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:09.869107 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:09.869231 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:09.869594 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:09.875931 18838 server_base.cc:1034] running on GCE node
I20240503 14:14:09.876811 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:09.878168 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:09.879467 18838 hybrid_clock.cc:648] HybridClock initialized: now 1714745649879392 us; error 68 us; skew 500 ppm
I20240503 14:14:09.879853 18838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:09.882870 18838 webserver.cc:460] Webserver started at http://127.10.145.6:38669/ using document root <none> and password file <none>
I20240503 14:14:09.883765 18838 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:09.883903 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:09.884274 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:09.887308 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "69cbbbedfe03437b8c95b7bb730dbe66"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-v549"
I20240503 14:14:09.888159 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "69cbbbedfe03437b8c95b7bb730dbe66"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-v549"
I20240503 14:14:09.894235 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20240503 14:14:09.898258 18852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:09.899924 18838 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:14:09.900135 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "69cbbbedfe03437b8c95b7bb730dbe66"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-v549"
I20240503 14:14:09.900370 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:09.932722 18838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:09.933702 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:09.933971 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:09.934978 18838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:09.936563 18838 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:09.936689 18838 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:09.936782 18838 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:09.936834 18838 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:09.967985 18838 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.6:36735
I20240503 14:14:09.968053 18964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.6:36735 every 8 connection(s)
I20240503 14:14:09.969483 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:14:09.970402 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 18838
I20240503 14:14:09.970603 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:14:09.974403 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.7:0
--local_ip_for_outbound_sockets=127.10.145.7
--webserver_interface=127.10.145.7
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33209
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:09.989045 18965 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:09.989437 18965 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:09.990425 18965 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:09.992115 18128 ts_manager.cc:194] Registered new tserver with Master: 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735)
I20240503 14:14:09.992976 18128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.6:34921
I20240503 14:14:09.999691 18705 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
W20240503 14:14:10.353910 18968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:10.354303 18968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:10.354476 18968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:10.362679 18968 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:10.362809 18968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:10.362987 18968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.7
I20240503 14:14:10.372876 18968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.10.145.7
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.7
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:10.374397 18968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:10.376441 18968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:10.389763 18974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:10.390117 18973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:10.390507 18976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:10.391064 18968 server_base.cc:1034] running on GCE node
I20240503 14:14:10.391880 18968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:10.393271 18968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:10.394572 18968 hybrid_clock.cc:648] HybridClock initialized: now 1714745650394492 us; error 85 us; skew 500 ppm
I20240503 14:14:10.394927 18968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:10.397521 18968 webserver.cc:460] Webserver started at http://127.10.145.7:33349/ using document root <none> and password file <none>
I20240503 14:14:10.398324 18968 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:10.398444 18968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:10.398770 18968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:10.401768 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a632094ba4554a4895d6c0b538b81af8"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-v549"
I20240503 14:14:10.402623 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a632094ba4554a4895d6c0b538b81af8"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-v549"
I20240503 14:14:10.408463 18968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240503 14:14:10.412286 18982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.413872 18968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:10.414053 18968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a632094ba4554a4895d6c0b538b81af8"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-v549"
I20240503 14:14:10.414269 18968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:10.438346 18968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:10.439348 18968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:10.439635 18968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:10.440635 18968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:10.442410 18968 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:10.442508 18968 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.442598 18968 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:10.442653 18968 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.473912 18968 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.7:38307
I20240503 14:14:10.473987 19094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.7:38307 every 8 connection(s)
I20240503 14:14:10.475324 18968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:14:10.485013 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 18968
I20240503 14:14:10.485209 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:14:10.488878 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.8:0
--local_ip_for_outbound_sockets=127.10.145.8
--webserver_interface=127.10.145.8
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33209
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:10.489176 19095 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:10.489528 19095 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:10.490586 19095 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:10.492108 18835 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
I20240503 14:14:10.492264 18128 ts_manager.cc:194] Registered new tserver with Master: a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307)
I20240503 14:14:10.493232 18128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.7:39797
W20240503 14:14:10.852341 19098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:10.852733 19098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:10.852870 19098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:10.861543 19098 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:10.861689 19098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:10.861855 19098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.8
I20240503 14:14:10.872328 19098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.10.145.8
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.8
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:10.874060 19098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:10.876139 19098 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:10.890240 19103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:10.890383 19104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:10.890671 19106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:10.891163 19098 server_base.cc:1034] running on GCE node
I20240503 14:14:10.891832 19098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:10.892879 19098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:10.894078 19098 hybrid_clock.cc:648] HybridClock initialized: now 1714745650894020 us; error 45 us; skew 500 ppm
I20240503 14:14:10.894412 19098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:10.896852 19098 webserver.cc:460] Webserver started at http://127.10.145.8:38929/ using document root <none> and password file <none>
I20240503 14:14:10.897697 19098 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:10.897815 19098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:10.898159 19098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:10.901034 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "80260781ae364d049c2500ff76a06ce8"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-v549"
I20240503 14:14:10.901921 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "80260781ae364d049c2500ff76a06ce8"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-v549"
I20240503 14:14:10.907568 19098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:14:10.911334 19112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.912916 19098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:10.913198 19098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "80260781ae364d049c2500ff76a06ce8"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-v549"
I20240503 14:14:10.913482 19098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:10.936087 19098 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:10.937040 19098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:10.937376 19098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:10.938448 19098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:10.940047 19098 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:10.940145 19098 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.940250 19098 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:10.940317 19098 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:10.971122 19098 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.8:44107
I20240503 14:14:10.971158 19224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.8:44107 every 8 connection(s)
I20240503 14:14:10.972623 19098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:14:10.983068 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 19098
I20240503 14:14:10.983289 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:14:10.986958 19225 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:10.987355 19225 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:10.988515 19225 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:10.990123 18128 ts_manager.cc:194] Registered new tserver with Master: 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107)
I20240503 14:14:10.990954 18128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.8:42009
I20240503 14:14:10.996423 18965 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
I20240503 14:14:11.001010 10820 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:14:11.018527 10820 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:14:11.018687 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 18708
I20240503 14:14:11.027004 18830 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:11.109052 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 18708
I20240503 14:14:11.122987 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 18838
I20240503 14:14:11.131354 18960 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:11.217965 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 18838
I20240503 14:14:11.231995 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 18968
I20240503 14:14:11.240458 19090 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:11.321957 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 18968
I20240503 14:14:11.335500 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 19098
I20240503 14:14:11.344275 19220 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:11.426103 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 19098
I20240503 14:14:15.192160 18128 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49596:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:15.234448 18639 tablet_service.cc:1467] Processing CreateTablet for tablet 8c337ef0405e4856bd2a608a6a602a49 (DEFAULT_TABLE table=rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.234414 18640 tablet_service.cc:1467] Processing CreateTablet for tablet d5b74777c90f4cb7937d0dfc54420467 (DEFAULT_TABLE table=rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.238188 18640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b74777c90f4cb7937d0dfc54420467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.239166 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c337ef0405e4856bd2a608a6a602a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.240839 18380 tablet_service.cc:1467] Processing CreateTablet for tablet d5b74777c90f4cb7937d0dfc54420467 (DEFAULT_TABLE table=rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.242537 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b74777c90f4cb7937d0dfc54420467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.243183 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 8c337ef0405e4856bd2a608a6a602a49 (DEFAULT_TABLE table=rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.243896 18379 tablet_service.cc:1467] Processing CreateTablet for tablet f0c1a7f5739f4fa59271254e501fb095 (DEFAULT_TABLE table=rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.245703 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c1a7f5739f4fa59271254e501fb095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.245941 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c337ef0405e4856bd2a608a6a602a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.251539 18509 tablet_service.cc:1467] Processing CreateTablet for tablet f0c1a7f5739f4fa59271254e501fb095 (DEFAULT_TABLE table=rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.252496 18249 tablet_service.cc:1467] Processing CreateTablet for tablet f0c1a7f5739f4fa59271254e501fb095 (DEFAULT_TABLE table=rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.254294 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c1a7f5739f4fa59271254e501fb095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.252744 18510 tablet_service.cc:1467] Processing CreateTablet for tablet d5b74777c90f4cb7937d0dfc54420467 (DEFAULT_TABLE table=rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.255298 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b74777c90f4cb7937d0dfc54420467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.258878 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c1a7f5739f4fa59271254e501fb095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.259323 18248 tablet_service.cc:1467] Processing CreateTablet for tablet 8c337ef0405e4856bd2a608a6a602a49 (DEFAULT_TABLE table=rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:15.259876 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c337ef0405e4856bd2a608a6a602a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:15.268354 19255 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:15.277607 19255 tablet_bootstrap.cc:654] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.280311 19254 tablet_bootstrap.cc:492] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:15.280916 19255 log.cc:824] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:15.282792 19256 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:15.284420 19254 tablet_bootstrap.cc:654] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.285560 19258 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:15.285996 19254 log.cc:824] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:15.287822 19256 tablet_bootstrap.cc:654] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.288863 19256 log.cc:824] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:15.289685 19258 tablet_bootstrap.cc:654] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.289856 19254 tablet_bootstrap.cc:492] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:15.290218 19254 ts_tablet_manager.cc:1367] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240503 14:14:15.291074 19258 log.cc:824] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:15.291779 19255 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:15.292198 19255 ts_tablet_manager.cc:1367] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.005s
I20240503 14:14:15.300076 19254 raft_consensus.cc:348] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:15.300562 19258 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:15.300565 19256 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:15.300670 19254 raft_consensus.cc:374] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.300796 19254 raft_consensus.cc:729] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.300891 19256 ts_tablet_manager.cc:1367] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.010s
I20240503 14:14:15.300889 19258 ts_tablet_manager.cc:1367] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240503 14:14:15.301647 19254 consensus_queue.cc:260] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:15.306030 19255 raft_consensus.cc:348] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:15.306725 19255 raft_consensus.cc:374] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.306880 19255 raft_consensus.cc:729] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.307749 19255 consensus_queue.cc:260] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:15.310348 19258 raft_consensus.cc:348] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:15.311024 19258 raft_consensus.cc:374] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.311192 19258 raft_consensus.cc:729] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.311453 19256 raft_consensus.cc:348] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:15.312032 19256 raft_consensus.cc:374] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.312186 19256 raft_consensus.cc:729] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.313030 19256 consensus_queue.cc:260] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:15.313386 19254 ts_tablet_manager.cc:1398] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.023s user 0.019s sys 0.004s
I20240503 14:14:15.313947 19254 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:15.316066 19254 tablet_bootstrap.cc:654] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.317848 19255 ts_tablet_manager.cc:1398] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.025s user 0.011s sys 0.007s
I20240503 14:14:15.318392 19255 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:15.320142 19255 tablet_bootstrap.cc:654] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.322948 19263 raft_consensus.cc:482] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:15.323307 19263 raft_consensus.cc:504] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:15.313055 19258 consensus_queue.cc:260] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:15.333642 19256 ts_tablet_manager.cc:1398] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.033s user 0.024s sys 0.007s
I20240503 14:14:15.334414 19256 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:15.336558 19256 tablet_bootstrap.cc:654] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.342254 19263 leader_election.cc:288] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:15.345467 19256 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:15.345826 19256 ts_tablet_manager.cc:1367] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240503 14:14:15.347151 19256 raft_consensus.cc:348] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:15.347589 19256 raft_consensus.cc:374] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.347728 19256 raft_consensus.cc:729] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.348119 19256 consensus_queue.cc:260] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:15.349129 19256 ts_tablet_manager.cc:1398] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:15.355226 19254 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:15.355538 19254 ts_tablet_manager.cc:1367] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.024s
I20240503 14:14:15.356782 19254 raft_consensus.cc:348] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:15.357113 19254 raft_consensus.cc:374] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.357259 19254 raft_consensus.cc:729] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.357704 19254 consensus_queue.cc:260] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:15.358325 19254 ts_tablet_manager.cc:1398] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:15.359377 19258 ts_tablet_manager.cc:1398] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.058s user 0.031s sys 0.011s
I20240503 14:14:15.359760 19255 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:15.359948 19258 tablet_bootstrap.cc:492] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:15.359982 19255 ts_tablet_manager.cc:1367] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.006s
I20240503 14:14:15.361035 19255 raft_consensus.cc:348] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:15.361340 19255 raft_consensus.cc:374] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.361451 19255 raft_consensus.cc:729] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.361799 19255 consensus_queue.cc:260] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:15.362252 19255 ts_tablet_manager.cc:1398] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:15.362571 19255 tablet_bootstrap.cc:492] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:15.364288 19255 tablet_bootstrap.cc:654] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.366178 19258 tablet_bootstrap.cc:654] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:15.368327 19258 tablet_bootstrap.cc:492] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:15.368496 19258 ts_tablet_manager.cc:1367] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 14:14:15.369607 19258 raft_consensus.cc:348] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:15.369912 19258 raft_consensus.cc:374] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.370069 19258 raft_consensus.cc:729] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.370129 19255 tablet_bootstrap.cc:492] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:15.370361 19255 ts_tablet_manager.cc:1367] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240503 14:14:15.370481 19258 consensus_queue.cc:260] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:15.371119 19258 ts_tablet_manager.cc:1398] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:15.371248 19255 raft_consensus.cc:348] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:15.371471 19255 raft_consensus.cc:374] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:15.371549 19255 raft_consensus.cc:729] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:15.371806 19255 consensus_queue.cc:260] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:15.372469 19255 ts_tablet_manager.cc:1398] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:15.372283 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c337ef0405e4856bd2a608a6a602a49" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:15.372855 18269 raft_consensus.cc:2454] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:15.373981 18333 leader_election.cc:302] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:15.374426 19263 raft_consensus.cc:2781] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:15.374616 19263 raft_consensus.cc:482] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:15.374758 19263 raft_consensus.cc:3037] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.376008 18660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c337ef0405e4856bd2a608a6a602a49" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:15.376492 18660 raft_consensus.cc:2454] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:15.377661 19263 raft_consensus.cc:504] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:15.378255 19263 leader_election.cc:288] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:15.378672 18660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c337ef0405e4856bd2a608a6a602a49" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:15.378796 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c337ef0405e4856bd2a608a6a602a49" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:15.378865 18660 raft_consensus.cc:3037] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.379000 18269 raft_consensus.cc:3037] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.381080 18660 raft_consensus.cc:2454] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:15.381137 18269 raft_consensus.cc:2454] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:15.381551 18333 leader_election.cc:302] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:15.381886 19263 raft_consensus.cc:2781] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:15.382290 19263 raft_consensus.cc:686] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Becoming Leader. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Running, Role: LEADER
I20240503 14:14:15.382833 19263 consensus_queue.cc:237] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:15.387993 18127 catalog_manager.cc:5484] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:15.415377 19263 raft_consensus.cc:482] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:15.415649 19263 raft_consensus.cc:504] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:15.416559 19263 leader_election.cc:288] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:15.417382 18660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b74777c90f4cb7937d0dfc54420467" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:15.417658 18660 raft_consensus.cc:2454] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:15.418144 18333 leader_election.cc:302] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:15.418489 19263 raft_consensus.cc:2781] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:15.418607 19263 raft_consensus.cc:482] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:15.418761 19263 raft_consensus.cc:3037] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.421149 19263 raft_consensus.cc:504] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:15.422029 19263 leader_election.cc:288] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Requested vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:15.422538 18660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b74777c90f4cb7937d0dfc54420467" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:15.422766 18660 raft_consensus.cc:3037] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.423514 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b74777c90f4cb7937d0dfc54420467" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:15.423542 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b74777c90f4cb7937d0dfc54420467" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:15.423995 18529 raft_consensus.cc:3037] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.425542 18660 raft_consensus.cc:2454] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:15.426043 18333 leader_election.cc:302] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:15.426455 19263 raft_consensus.cc:2781] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:15.426563 18529 raft_consensus.cc:2454] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:15.426666 19263 raft_consensus.cc:686] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Becoming Leader. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Running, Role: LEADER
I20240503 14:14:15.427053 19263 consensus_queue.cc:237] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:15.429843 18127 catalog_manager.cc:5484] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:15.457185 19263 raft_consensus.cc:482] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:15.457437 19263 raft_consensus.cc:504] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:15.458168 19263 leader_election.cc:288] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:15.458765 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1a7f5739f4fa59271254e501fb095" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:15.458765 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1a7f5739f4fa59271254e501fb095" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:15.459060 18529 raft_consensus.cc:2454] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:15.459061 18269 raft_consensus.cc:2454] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:15.459524 18333 leader_election.cc:302] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:15.459861 19263 raft_consensus.cc:2781] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:15.459968 19263 raft_consensus.cc:482] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:15.460059 19263 raft_consensus.cc:3037] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.462003 19263 raft_consensus.cc:504] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:15.462626 19263 leader_election.cc:288] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Requested vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:15.463230 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1a7f5739f4fa59271254e501fb095" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:15.463234 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1a7f5739f4fa59271254e501fb095" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:15.463478 18269 raft_consensus.cc:3037] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.463481 18529 raft_consensus.cc:3037] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:15.466348 18269 raft_consensus.cc:2454] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:15.466348 18529 raft_consensus.cc:2454] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:15.466917 18336 leader_election.cc:302] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:15.467248 19263 raft_consensus.cc:2781] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:15.467413 19263 raft_consensus.cc:686] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Becoming Leader. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Running, Role: LEADER
I20240503 14:14:15.467757 19263 consensus_queue.cc:237] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:15.470590 18127 catalog_manager.cc:5484] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } }
W20240503 14:14:15.486594 18706 tablet.cc:2369] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:14:15.494617 18446 tablet.cc:2369] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:14:15.503875 18576 tablet.cc:2369] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:14:15.518395 18315 tablet.cc:2369] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:14:15.774998 19263 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:15.784971 19263 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:15.843757 19263 consensus_queue.cc:1035] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:15.851889 19273 consensus_queue.cc:1035] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:15.874650 19277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:15.875137 19277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:15.883119 19277 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 14:14:15.928404 19273 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:15.933544 19302 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 252.859 ms
time per row: 0.12643 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:14:17.172369 18124 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49596:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:17.186071 18509 tablet_service.cc:1467] Processing CreateTablet for tablet 19b1b6be3bda47829f53765282eae9f9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.186614 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b1b6be3bda47829f53765282eae9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.186645 18640 tablet_service.cc:1467] Processing CreateTablet for tablet 19b1b6be3bda47829f53765282eae9f9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.186733 18249 tablet_service.cc:1467] Processing CreateTablet for tablet 7570d5bce0e5481d8dded653617be5a7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.187126 18640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b1b6be3bda47829f53765282eae9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.187144 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7570d5bce0e5481d8dded653617be5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.186653 18639 tablet_service.cc:1467] Processing CreateTablet for tablet 7570d5bce0e5481d8dded653617be5a7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.189270 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7570d5bce0e5481d8dded653617be5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.190168 18247 tablet_service.cc:1467] Processing CreateTablet for tablet d907591e3c9740e8b5eee42f8d41996b (DEFAULT_TABLE table=rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.190663 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d907591e3c9740e8b5eee42f8d41996b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.194298 18510 tablet_service.cc:1467] Processing CreateTablet for tablet 7570d5bce0e5481d8dded653617be5a7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.194823 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7570d5bce0e5481d8dded653617be5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.197357 18248 tablet_service.cc:1467] Processing CreateTablet for tablet 19b1b6be3bda47829f53765282eae9f9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.197228 19341 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:17.197942 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b1b6be3bda47829f53765282eae9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.199684 19341 tablet_bootstrap.cc:654] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.202406 18508 tablet_service.cc:1467] Processing CreateTablet for tablet d907591e3c9740e8b5eee42f8d41996b (DEFAULT_TABLE table=rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.202972 18508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d907591e3c9740e8b5eee42f8d41996b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.202910 18379 tablet_service.cc:1467] Processing CreateTablet for tablet d907591e3c9740e8b5eee42f8d41996b (DEFAULT_TABLE table=rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:17.203403 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d907591e3c9740e8b5eee42f8d41996b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:17.206473 19341 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:17.207569 19341 ts_tablet_manager.cc:1367] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 14:14:17.208787 19341 raft_consensus.cc:348] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.201857 19343 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:17.210587 19341 raft_consensus.cc:374] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.210919 19341 raft_consensus.cc:729] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.211536 19341 consensus_queue.cc:260] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.212327 19343 tablet_bootstrap.cc:654] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.215570 19342 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:17.217710 19342 tablet_bootstrap.cc:654] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.219074 19341 ts_tablet_manager.cc:1398] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20240503 14:14:17.219537 19341 tablet_bootstrap.cc:492] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:17.221485 19341 tablet_bootstrap.cc:654] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.222100 19343 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:17.222312 19343 ts_tablet_manager.cc:1367] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.005s
I20240503 14:14:17.223265 19343 raft_consensus.cc:348] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:17.223546 19343 raft_consensus.cc:374] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.223661 19343 raft_consensus.cc:729] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.223935 19343 consensus_queue.cc:260] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:17.228731 19343 ts_tablet_manager.cc:1398] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 14:14:17.229205 19343 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:17.231385 19343 tablet_bootstrap.cc:654] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.233247 19341 tablet_bootstrap.cc:492] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:17.233604 19341 ts_tablet_manager.cc:1367] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20240503 14:14:17.235283 19341 raft_consensus.cc:348] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.235684 19341 raft_consensus.cc:374] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.235813 19341 raft_consensus.cc:729] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.236204 19341 consensus_queue.cc:260] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.236863 19341 ts_tablet_manager.cc:1398] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:14:17.237236 19341 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:17.238095 19352 tablet_bootstrap.cc:492] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:17.239655 19341 tablet_bootstrap.cc:654] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.240620 19352 tablet_bootstrap.cc:654] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.245033 19343 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:17.245419 19343 ts_tablet_manager.cc:1367] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.009s
I20240503 14:14:17.245662 19341 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:17.245918 19341 ts_tablet_manager.cc:1367] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240503 14:14:17.246491 19343 raft_consensus.cc:348] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.246876 19343 raft_consensus.cc:374] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.246807 19341 raft_consensus.cc:348] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:17.247027 19343 raft_consensus.cc:729] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.247140 19341 raft_consensus.cc:374] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.247300 19341 raft_consensus.cc:729] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.247422 19343 consensus_queue.cc:260] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.247733 19341 consensus_queue.cc:260] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:17.248292 19343 ts_tablet_manager.cc:1398] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:14:17.249027 19341 ts_tablet_manager.cc:1398] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:17.249164 19342 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:17.249401 19342 ts_tablet_manager.cc:1367] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.021s
I20240503 14:14:17.250316 19342 raft_consensus.cc:348] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:17.250546 19342 raft_consensus.cc:374] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.250679 19342 raft_consensus.cc:729] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.250917 19342 consensus_queue.cc:260] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:17.251684 19342 ts_tablet_manager.cc:1398] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:17.252024 19342 tablet_bootstrap.cc:492] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:17.253760 19342 tablet_bootstrap.cc:654] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.254689 19352 tablet_bootstrap.cc:492] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:17.254943 19352 ts_tablet_manager.cc:1367] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 14:14:17.255884 19352 raft_consensus.cc:348] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.256083 19352 raft_consensus.cc:374] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.256202 19352 raft_consensus.cc:729] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.256440 19352 consensus_queue.cc:260] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.257035 19352 ts_tablet_manager.cc:1398] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:17.258682 19342 tablet_bootstrap.cc:492] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:17.258884 19342 ts_tablet_manager.cc:1367] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240503 14:14:17.259625 19342 raft_consensus.cc:348] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.259871 19342 raft_consensus.cc:374] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.259948 19342 raft_consensus.cc:729] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.260179 19342 consensus_queue.cc:260] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.260660 19342 ts_tablet_manager.cc:1398] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:17.260926 19342 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:17.262552 19342 tablet_bootstrap.cc:654] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:17.264444 19342 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:17.264604 19342 ts_tablet_manager.cc:1367] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:14:17.265255 19342 raft_consensus.cc:348] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.265470 19342 raft_consensus.cc:374] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:17.265532 19342 raft_consensus.cc:729] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:17.265724 19342 consensus_queue.cc:260] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.266170 19342 ts_tablet_manager.cc:1398] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:14:17.381121 19350 raft_consensus.cc:482] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:17.381364 19350 raft_consensus.cc:504] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:17.383038 19350 leader_election.cc:288] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:17.384692 19346 raft_consensus.cc:482] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:17.384907 19346 raft_consensus.cc:504] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:17.386778 19346 leader_election.cc:288] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:17.393008 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7570d5bce0e5481d8dded653617be5a7" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:17.393360 18529 raft_consensus.cc:2454] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:17.393610 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7570d5bce0e5481d8dded653617be5a7" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:17.393848 18268 raft_consensus.cc:2454] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:17.394026 18596 leader_election.cc:302] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:17.394544 19350 raft_consensus.cc:2781] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:17.394698 19350 raft_consensus.cc:482] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:17.394796 19350 raft_consensus.cc:3037] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.397946 19350 raft_consensus.cc:504] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:17.399363 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7570d5bce0e5481d8dded653617be5a7" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:17.399652 18268 raft_consensus.cc:3037] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.399963 19350 leader_election.cc:288] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Requested vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:17.399890 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7570d5bce0e5481d8dded653617be5a7" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:17.400125 18529 raft_consensus.cc:3037] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.400462 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7570d5bce0e5481d8dded653617be5a7" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:17.402702 18529 raft_consensus.cc:2454] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:17.402680 18268 raft_consensus.cc:2454] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:17.403164 18596 leader_election.cc:302] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:17.403543 19350 raft_consensus.cc:2781] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:17.403779 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7570d5bce0e5481d8dded653617be5a7" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:17.404023 19350 raft_consensus.cc:686] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Becoming Leader. State: Replica: 1c72210d694344d4b4604140088a6471, State: Running, Role: LEADER
I20240503 14:14:17.404489 19350 consensus_queue.cc:237] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:17.407441 18463 leader_election.cc:302] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722; no voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838
I20240503 14:14:17.407974 19346 raft_consensus.cc:2726] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:14:17.408468 18122 catalog_manager.cc:5484] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c72210d694344d4b4604140088a6471 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:17.424127 19345 raft_consensus.cc:482] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:17.424350 19345 raft_consensus.cc:504] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.425158 19345 leader_election.cc:288] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:17.425693 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d907591e3c9740e8b5eee42f8d41996b" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:17.425709 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d907591e3c9740e8b5eee42f8d41996b" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:17.425940 18529 raft_consensus.cc:2454] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:17.425947 18268 raft_consensus.cc:2454] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:17.426434 18336 leader_election.cc:302] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:17.426803 19345 raft_consensus.cc:2781] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:17.426925 19345 raft_consensus.cc:482] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:17.427018 19345 raft_consensus.cc:3037] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.428951 19345 raft_consensus.cc:504] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.429670 19345 leader_election.cc:288] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Requested vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:17.430218 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d907591e3c9740e8b5eee42f8d41996b" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:17.430224 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d907591e3c9740e8b5eee42f8d41996b" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:17.430426 18529 raft_consensus.cc:3037] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.430437 18268 raft_consensus.cc:3037] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.432366 18529 raft_consensus.cc:2454] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:17.432404 18268 raft_consensus.cc:2454] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:17.432900 18333 leader_election.cc:302] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:17.433295 19345 raft_consensus.cc:2781] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:17.433441 19345 raft_consensus.cc:686] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Becoming Leader. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Running, Role: LEADER
I20240503 14:14:17.433719 19345 consensus_queue.cc:237] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.436401 18122 catalog_manager.cc:5484] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:17.555011 19346 raft_consensus.cc:482] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:17.555254 19346 raft_consensus.cc:504] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.555907 19346 leader_election.cc:288] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:17.556576 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b1b6be3bda47829f53765282eae9f9" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:17.556574 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b1b6be3bda47829f53765282eae9f9" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:17.556842 18268 raft_consensus.cc:2454] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 0.
I20240503 14:14:17.556864 18659 raft_consensus.cc:2454] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 0.
I20240503 14:14:17.557317 18463 leader_election.cc:302] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:17.557596 19346 raft_consensus.cc:2781] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:17.557736 19346 raft_consensus.cc:482] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:17.557833 19346 raft_consensus.cc:3037] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.559948 19346 raft_consensus.cc:504] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.560564 19346 leader_election.cc:288] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 election: Requested vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:17.561144 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b1b6be3bda47829f53765282eae9f9" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:17.561167 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b1b6be3bda47829f53765282eae9f9" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:17.561393 18268 raft_consensus.cc:3037] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.561412 18659 raft_consensus.cc:3037] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:17.564296 18268 raft_consensus.cc:2454] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 1.
I20240503 14:14:17.564285 18659 raft_consensus.cc:2454] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 1.
I20240503 14:14:17.564864 18463 leader_election.cc:302] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:17.565207 19346 raft_consensus.cc:2781] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:17.565708 19346 raft_consensus.cc:686] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Becoming Leader. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Running, Role: LEADER
I20240503 14:14:17.566238 19346 consensus_queue.cc:237] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:17.569969 18122 catalog_manager.cc:5484] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3). New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:17.811676 19350 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:17.821182 19350 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:17.846400 19345 consensus_queue.cc:1035] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:17.853591 19380 consensus_queue.cc:1035] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:17.962702 19371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:17.963060 19371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:17.970652 19371 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 14:14:18.041980 19346 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:18.049372 19369 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 235.645 ms
time per row: 0.117822 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:14:19.244058 18121 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49596:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:19.254964 18639 tablet_service.cc:1467] Processing CreateTablet for tablet 5a23a38ec48148b19bf8102a944ab11e (DEFAULT_TABLE table=rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:19.255528 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a23a38ec48148b19bf8102a944ab11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:19.255568 18249 tablet_service.cc:1467] Processing CreateTablet for tablet 5a23a38ec48148b19bf8102a944ab11e (DEFAULT_TABLE table=rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:19.256027 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a23a38ec48148b19bf8102a944ab11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:19.258431 18248 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc67ebb58a246d9a8c47f0838a4831a (DEFAULT_TABLE table=rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:19.258407 18379 tablet_service.cc:1467] Processing CreateTablet for tablet 5a23a38ec48148b19bf8102a944ab11e (DEFAULT_TABLE table=rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:19.258533 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 7c63b35eef494f0e8928c4ca0f74ec32 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:19.258878 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a23a38ec48148b19bf8102a944ab11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:19.258885 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc67ebb58a246d9a8c47f0838a4831a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:19.259266 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc67ebb58a246d9a8c47f0838a4831a (DEFAULT_TABLE table=rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:19.259685 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc67ebb58a246d9a8c47f0838a4831a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:19.261780 18510 tablet_service.cc:1467] Processing CreateTablet for tablet 7c63b35eef494f0e8928c4ca0f74ec32 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:19.262238 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c63b35eef494f0e8928c4ca0f74ec32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:19.265697 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c63b35eef494f0e8928c4ca0f74ec32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:19.271283 19440 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:19.273231 19440 tablet_bootstrap.cc:654] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:19.274226 18640 tablet_service.cc:1467] Processing CreateTablet for tablet 7c63b35eef494f0e8928c4ca0f74ec32 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:19.274829 18640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c63b35eef494f0e8928c4ca0f74ec32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:19.277966 18638 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc67ebb58a246d9a8c47f0838a4831a (DEFAULT_TABLE table=rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:19.278568 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc67ebb58a246d9a8c47f0838a4831a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:19.289897 19442 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:19.293354 19442 tablet_bootstrap.cc:654] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:19.297502 19442 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:19.298863 19439 tablet_bootstrap.cc:492] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:19.299808 19442 ts_tablet_manager.cc:1367] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20240503 14:14:19.300350 19440 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:19.300616 19440 ts_tablet_manager.cc:1367] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.004s
I20240503 14:14:19.300949 19443 tablet_bootstrap.cc:492] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:19.301551 19440 raft_consensus.cc:348] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:19.301756 19440 raft_consensus.cc:374] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:19.301874 19440 raft_consensus.cc:729] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:19.302019 19442 raft_consensus.cc:348] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.302126 19440 consensus_queue.cc:260] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:19.302270 19442 raft_consensus.cc:374] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:19.302800 19440 ts_tablet_manager.cc:1398] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:19.303678 19443 tablet_bootstrap.cc:654] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:19.304040 19439 tablet_bootstrap.cc:654] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:19.304956 19442 raft_consensus.cc:729] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:19.305332 19442 consensus_queue.cc:260] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.307250 19442 ts_tablet_manager.cc:1398] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:14:19.307672 19442 tablet_bootstrap.cc:492] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:19.310133 19442 tablet_bootstrap.cc:654] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:19.312561 19439 tablet_bootstrap.cc:492] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:19.312726 19442 tablet_bootstrap.cc:492] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:19.312865 19443 tablet_bootstrap.cc:492] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:19.312948 19442 ts_tablet_manager.cc:1367] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 14:14:19.313127 19443 ts_tablet_manager.cc:1367] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 14:14:19.313488 19439 ts_tablet_manager.cc:1367] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 14:14:19.313841 19442 raft_consensus.cc:348] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.314137 19442 raft_consensus.cc:374] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:19.314214 19442 raft_consensus.cc:729] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:19.314101 19443 raft_consensus.cc:348] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.314394 19443 raft_consensus.cc:374] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:19.314522 19443 raft_consensus.cc:729] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:19.314409 19439 raft_consensus.cc:348] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.314488 19442 consensus_queue.cc:260] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.314635 19439 raft_consensus.cc:374] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:19.314723 19439 raft_consensus.cc:729] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:19.315091 19442 ts_tablet_manager.cc:1398] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:19.315006 19439 consensus_queue.cc:260] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.315878 19439 ts_tablet_manager.cc:1398] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:19.316027 19442 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:19.314822 19443 consensus_queue.cc:260] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.317247 19443 ts_tablet_manager.cc:1398] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:14:19.317692 19439 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:19.317780 19442 tablet_bootstrap.cc:654] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:19.318976 19443 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:19.319669 19439 tablet_bootstrap.cc:654] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:19.322340 19443 tablet_bootstrap.cc:654] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:19.327870 19439 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:19.328143 19443 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:19.328130 19439 ts_tablet_manager.cc:1367] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240503 14:14:19.328336 19443 ts_tablet_manager.cc:1367] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 14:14:19.328784 19442 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:19.328943 19442 ts_tablet_manager.cc:1367] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240503 14:14:19.329067 19439 raft_consensus.cc:348] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.329151 19443 raft_consensus.cc:348] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:19.329382 19439 raft_consensus.cc:374] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:19.329478 19443 raft_consensus.cc:374] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:19.329521 19439 raft_consensus.cc:729] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:19.329574 19443 raft_consensus.cc:729] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:19.329658 19442 raft_consensus.cc:348] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:19.329870 19442 raft_consensus.cc:374] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:19.329964 19442 raft_consensus.cc:729] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:19.329900 19439 consensus_queue.cc:260] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.329921 19443 consensus_queue.cc:260] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:19.330201 19442 consensus_queue.cc:260] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:19.330650 19439 ts_tablet_manager.cc:1398] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:14:19.330914 19442 ts_tablet_manager.cc:1398] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:19.331593 19443 ts_tablet_manager.cc:1398] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:19.331940 19443 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:19.333722 19443 tablet_bootstrap.cc:654] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:19.336702 19443 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:19.336869 19443 ts_tablet_manager.cc:1367] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:14:19.337764 19443 raft_consensus.cc:348] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.337988 19443 raft_consensus.cc:374] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:19.338093 19443 raft_consensus.cc:729] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:19.338311 19443 consensus_queue.cc:260] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.338848 19443 ts_tablet_manager.cc:1398] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:19.433887 19369 raft_consensus.cc:482] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:19.434118 19369 raft_consensus.cc:504] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:19.434870 19369 leader_election.cc:288] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:19.435657 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c63b35eef494f0e8928c4ca0f74ec32" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:19.436010 18659 raft_consensus.cc:2454] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 0.
I20240503 14:14:19.436579 18463 leader_election.cc:302] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:19.436915 19369 raft_consensus.cc:2781] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:19.437024 19369 raft_consensus.cc:482] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:19.437139 19369 raft_consensus.cc:3037] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:19.439241 19369 raft_consensus.cc:504] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:19.439967 19369 leader_election.cc:288] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:19.440551 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c63b35eef494f0e8928c4ca0f74ec32" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:19.440814 18659 raft_consensus.cc:3037] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:19.442781 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c63b35eef494f0e8928c4ca0f74ec32" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:19.442834 18400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c63b35eef494f0e8928c4ca0f74ec32" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:19.443161 18399 raft_consensus.cc:3037] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:19.443686 18659 raft_consensus.cc:2454] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 1.
I20240503 14:14:19.444208 18463 leader_election.cc:302] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:19.444563 19369 raft_consensus.cc:2781] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:19.445070 19369 raft_consensus.cc:686] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Becoming Leader. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Running, Role: LEADER
I20240503 14:14:19.445447 19369 consensus_queue.cc:237] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:19.445720 18399 raft_consensus.cc:2454] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 1.
I20240503 14:14:19.448616 18121 catalog_manager.cc:5484] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3). New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:19.504981 19434 raft_consensus.cc:482] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:19.505192 19434 raft_consensus.cc:504] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.506256 19434 leader_election.cc:288] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:19.507761 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc67ebb58a246d9a8c47f0838a4831a" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:19.508054 18268 raft_consensus.cc:2454] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:19.508630 18593 leader_election.cc:302] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:19.508952 19434 raft_consensus.cc:2781] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:19.509059 19434 raft_consensus.cc:482] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:19.509147 19434 raft_consensus.cc:3037] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:19.511744 19434 raft_consensus.cc:504] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.513126 19434 leader_election.cc:288] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:19.513129 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc67ebb58a246d9a8c47f0838a4831a" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:19.513425 18268 raft_consensus.cc:3037] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:19.513597 19345 raft_consensus.cc:482] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:19.513756 19345 raft_consensus.cc:504] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.514560 19345 leader_election.cc:288] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:19.515065 18400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc67ebb58a246d9a8c47f0838a4831a" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:19.514990 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc67ebb58a246d9a8c47f0838a4831a" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:19.515272 18400 raft_consensus.cc:3037] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:19.515480 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a23a38ec48148b19bf8102a944ab11e" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:19.515715 18659 raft_consensus.cc:2454] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:19.516299 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a23a38ec48148b19bf8102a944ab11e" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:19.516537 18269 raft_consensus.cc:2454] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:19.517040 18333 leader_election.cc:302] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:19.517220 18268 raft_consensus.cc:2454] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:19.517657 18593 leader_election.cc:302] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:19.517756 19345 raft_consensus.cc:2781] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:19.517864 19345 raft_consensus.cc:482] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:19.517990 19345 raft_consensus.cc:3037] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:19.518077 19434 raft_consensus.cc:2781] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:19.518226 18400 raft_consensus.cc:2454] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:19.519374 19434 raft_consensus.cc:686] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Becoming Leader. State: Replica: 1c72210d694344d4b4604140088a6471, State: Running, Role: LEADER
I20240503 14:14:19.519790 19434 consensus_queue.cc:237] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.521180 19345 raft_consensus.cc:504] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.521941 19345 leader_election.cc:288] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:19.522625 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a23a38ec48148b19bf8102a944ab11e" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:19.522881 18659 raft_consensus.cc:3037] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:19.522858 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a23a38ec48148b19bf8102a944ab11e" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:19.523098 18268 raft_consensus.cc:3037] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:19.523536 18121 catalog_manager.cc:5484] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c72210d694344d4b4604140088a6471 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:19.525635 18268 raft_consensus.cc:2454] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:19.526494 18333 leader_election.cc:302] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:19.526867 19345 raft_consensus.cc:2781] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:19.526839 18659 raft_consensus.cc:2454] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:19.528039 19345 raft_consensus.cc:686] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Becoming Leader. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Running, Role: LEADER
I20240503 14:14:19.528342 19345 consensus_queue.cc:237] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:19.531169 18121 catalog_manager.cc:5484] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:19.840574 19369 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:19.846293 19369 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:19.960311 19462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:19.960675 19462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:19.968465 19462 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:14:19.970583 19434 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:19.976107 19434 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:19.994208 19345 consensus_queue.cc:1035] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:20.009655 19460 consensus_queue.cc:1035] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 267.239 ms
time per row: 0.133619 ms
I20240503 14:14:20.407713 18124 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49596:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:20.417865 18510 tablet_service.cc:1467] Processing CreateTablet for tablet eedc3cfb2dff4f9084446d89b9b3c0f4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:20.418385 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedc3cfb2dff4f9084446d89b9b3c0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.418370 18378 tablet_service.cc:1467] Processing CreateTablet for tablet eedc3cfb2dff4f9084446d89b9b3c0f4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:20.418840 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedc3cfb2dff4f9084446d89b9b3c0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.418855 18639 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9705f607a94835ad7e6c20668b80ad (DEFAULT_TABLE table=rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:20.418972 18249 tablet_service.cc:1467] Processing CreateTablet for tablet eedc3cfb2dff4f9084446d89b9b3c0f4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:20.419257 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9705f607a94835ad7e6c20668b80ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.419399 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedc3cfb2dff4f9084446d89b9b3c0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.425824 18248 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9705f607a94835ad7e6c20668b80ad (DEFAULT_TABLE table=rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:20.426363 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9705f607a94835ad7e6c20668b80ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.430356 18509 tablet_service.cc:1467] Processing CreateTablet for tablet 7f9705f607a94835ad7e6c20668b80ad (DEFAULT_TABLE table=rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:20.430881 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9705f607a94835ad7e6c20668b80ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.431583 18638 tablet_service.cc:1467] Processing CreateTablet for tablet 3de72df2a82e445f9aa783bc741fa996 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:20.432044 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3de72df2a82e445f9aa783bc741fa996. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.432072 18379 tablet_service.cc:1467] Processing CreateTablet for tablet 3de72df2a82e445f9aa783bc741fa996 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:20.432457 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3de72df2a82e445f9aa783bc741fa996. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.438089 19531 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:20.440603 19531 tablet_bootstrap.cc:654] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:20.445556 18509 tablet_service.cc:1467] Processing CreateTablet for tablet 3de72df2a82e445f9aa783bc741fa996 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:20.446343 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3de72df2a82e445f9aa783bc741fa996. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.447185 19533 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:20.449566 19533 tablet_bootstrap.cc:654] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:20.454216 19535 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:20.457084 19535 tablet_bootstrap.cc:654] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:20.460255 19532 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:20.461992 19532 tablet_bootstrap.cc:654] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:20.462436 19535 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:20.462620 19535 ts_tablet_manager.cc:1367] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:14:20.463550 19535 raft_consensus.cc:348] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.463794 19535 raft_consensus.cc:374] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:20.463877 19535 raft_consensus.cc:729] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.464116 19535 consensus_queue.cc:260] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.465816 19533 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:20.465984 19532 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:20.465994 19533 ts_tablet_manager.cc:1367] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.013s
I20240503 14:14:20.466145 19535 ts_tablet_manager.cc:1398] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:20.466145 19532 ts_tablet_manager.cc:1367] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:14:20.466470 19535 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:20.466884 19533 raft_consensus.cc:348] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.467111 19533 raft_consensus.cc:374] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:20.466989 19532 raft_consensus.cc:348] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:20.467198 19532 raft_consensus.cc:374] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:20.467201 19533 raft_consensus.cc:729] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.467281 19532 raft_consensus.cc:729] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.467439 19533 consensus_queue.cc:260] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.467548 19532 consensus_queue.cc:260] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:20.468168 19535 tablet_bootstrap.cc:654] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:20.468528 19533 ts_tablet_manager.cc:1398] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:20.468940 19533 tablet_bootstrap.cc:492] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:20.469437 19532 ts_tablet_manager.cc:1398] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:20.469786 19532 tablet_bootstrap.cc:492] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:20.470849 19533 tablet_bootstrap.cc:654] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:20.472030 19532 tablet_bootstrap.cc:654] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:20.477645 19532 tablet_bootstrap.cc:492] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:20.477944 19532 ts_tablet_manager.cc:1367] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240503 14:14:20.479290 19532 raft_consensus.cc:348] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.480135 19532 raft_consensus.cc:374] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:20.480275 19532 raft_consensus.cc:729] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.480695 19532 consensus_queue.cc:260] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.481411 19531 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:20.481633 19531 ts_tablet_manager.cc:1367] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.013s
I20240503 14:14:20.482033 19532 ts_tablet_manager.cc:1398] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:14:20.483660 19533 tablet_bootstrap.cc:492] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:20.483877 19533 ts_tablet_manager.cc:1367] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20240503 14:14:20.484499 19531 raft_consensus.cc:348] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:20.484728 19531 raft_consensus.cc:374] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:20.484830 19531 raft_consensus.cc:729] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.484719 19533 raft_consensus.cc:348] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.485096 19533 raft_consensus.cc:374] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:20.485213 19533 raft_consensus.cc:729] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.485108 19531 consensus_queue.cc:260] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:20.485757 19531 ts_tablet_manager.cc:1398] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:20.485775 19533 consensus_queue.cc:260] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.486055 19531 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:20.486645 19533 ts_tablet_manager.cc:1398] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 14:14:20.487998 19531 tablet_bootstrap.cc:654] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:20.490168 19535 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:20.490448 19535 ts_tablet_manager.cc:1367] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.015s
I20240503 14:14:20.491407 19535 raft_consensus.cc:348] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:20.491636 19535 raft_consensus.cc:374] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:20.491719 19535 raft_consensus.cc:729] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.492128 19535 consensus_queue.cc:260] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:20.492697 19535 ts_tablet_manager.cc:1398] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:20.497884 19531 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:20.498111 19531 ts_tablet_manager.cc:1367] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 14:14:20.498944 19531 raft_consensus.cc:348] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.499186 19531 raft_consensus.cc:374] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:20.499271 19531 raft_consensus.cc:729] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.499588 19531 consensus_queue.cc:260] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.500144 19531 ts_tablet_manager.cc:1398] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:20.500608 19531 tablet_bootstrap.cc:492] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:20.502804 19531 tablet_bootstrap.cc:654] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:20.505128 19531 tablet_bootstrap.cc:492] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:20.505333 19531 ts_tablet_manager.cc:1367] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:14:20.506255 19531 raft_consensus.cc:348] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.506568 19531 raft_consensus.cc:374] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:20.506690 19531 raft_consensus.cc:729] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.507318 19531 consensus_queue.cc:260] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.508073 19531 ts_tablet_manager.cc:1398] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:14:20.623495 19369 raft_consensus.cc:482] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:20.623773 19369 raft_consensus.cc:504] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.624580 19369 leader_election.cc:288] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:20.625183 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9705f607a94835ad7e6c20668b80ad" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:20.625205 18658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9705f607a94835ad7e6c20668b80ad" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:20.625509 18268 raft_consensus.cc:2454] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 0.
I20240503 14:14:20.625510 18658 raft_consensus.cc:2454] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 0.
I20240503 14:14:20.625983 18463 leader_election.cc:302] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:20.626394 19369 raft_consensus.cc:2781] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:20.626521 19369 raft_consensus.cc:482] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:20.626618 19369 raft_consensus.cc:3037] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:20.628633 19369 raft_consensus.cc:504] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.629256 19369 leader_election.cc:288] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 election: Requested vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:20.629848 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9705f607a94835ad7e6c20668b80ad" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:20.629921 18658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f9705f607a94835ad7e6c20668b80ad" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:20.630100 18268 raft_consensus.cc:3037] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:20.630112 18658 raft_consensus.cc:3037] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:20.632043 18658 raft_consensus.cc:2454] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 1.
I20240503 14:14:20.632122 18268 raft_consensus.cc:2454] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 1.
I20240503 14:14:20.632505 18463 leader_election.cc:302] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:20.632859 19369 raft_consensus.cc:2781] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:20.633020 19369 raft_consensus.cc:686] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Becoming Leader. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Running, Role: LEADER
I20240503 14:14:20.633442 19369 consensus_queue.cc:237] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.636643 18122 catalog_manager.cc:5484] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3). New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:20.654034 19472 raft_consensus.cc:482] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:20.654282 19472 raft_consensus.cc:504] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:20.655939 19472 leader_election.cc:288] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:20.662456 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedc3cfb2dff4f9084446d89b9b3c0f4" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:20.662402 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedc3cfb2dff4f9084446d89b9b3c0f4" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:20.662703 18399 raft_consensus.cc:2454] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:20.662720 18530 raft_consensus.cc:2454] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:20.663305 18204 leader_election.cc:302] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:20.663750 19472 raft_consensus.cc:2781] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:20.663897 19472 raft_consensus.cc:482] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:20.664016 19472 raft_consensus.cc:3037] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:20.666375 19472 raft_consensus.cc:504] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:20.667018 19472 leader_election.cc:288] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:20.667639 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedc3cfb2dff4f9084446d89b9b3c0f4" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:20.667634 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eedc3cfb2dff4f9084446d89b9b3c0f4" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:20.667861 18399 raft_consensus.cc:3037] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:20.667876 18530 raft_consensus.cc:3037] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:20.670745 18399 raft_consensus.cc:2454] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:20.670737 18530 raft_consensus.cc:2454] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:20.671288 18204 leader_election.cc:302] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:20.671707 19472 raft_consensus.cc:2781] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:20.672179 19472 raft_consensus.cc:686] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:20.672675 19472 consensus_queue.cc:237] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:20.676542 18122 catalog_manager.cc:5484] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:20.803772 19458 raft_consensus.cc:482] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:20.803964 19458 raft_consensus.cc:504] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.804677 19458 leader_election.cc:288] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:20.805157 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3de72df2a82e445f9aa783bc741fa996" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:20.805254 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3de72df2a82e445f9aa783bc741fa996" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:20.805426 18399 raft_consensus.cc:2454] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:20.805534 18530 raft_consensus.cc:2454] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:20.805908 18595 leader_election.cc:302] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:20.806196 19458 raft_consensus.cc:2781] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:20.806298 19458 raft_consensus.cc:482] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:20.806382 19458 raft_consensus.cc:3037] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:20.808264 19458 raft_consensus.cc:504] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.808810 19458 leader_election.cc:288] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:20.809388 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3de72df2a82e445f9aa783bc741fa996" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:20.809409 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3de72df2a82e445f9aa783bc741fa996" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:20.809603 18530 raft_consensus.cc:3037] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:20.809636 18399 raft_consensus.cc:3037] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:20.812345 18399 raft_consensus.cc:2454] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:20.812356 18530 raft_consensus.cc:2454] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:20.812829 18596 leader_election.cc:302] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:20.813139 19458 raft_consensus.cc:2781] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:20.813500 19458 raft_consensus.cc:686] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Becoming Leader. State: Replica: 1c72210d694344d4b4604140088a6471, State: Running, Role: LEADER
I20240503 14:14:20.813795 19458 consensus_queue.cc:237] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:20.816448 18122 catalog_manager.cc:5484] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c72210d694344d4b4604140088a6471 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:21.033402 19369 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:21.039557 19455 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:21.209708 19458 consensus_queue.cc:1035] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:21.215764 19549 consensus_queue.cc:1035] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:21.252235 19472 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:21.261773 19472 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:21.282271 19551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:21.282790 19551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:21.291018 19551 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 258.19 ms
time per row: 0.129095 ms
-----------------------------------------------------
Suppressions used:
count bytes template
12 3028 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:14:22.570423 18124 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49596:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:22.580444 18248 tablet_service.cc:1467] Processing CreateTablet for tablet c18cad62ab6443cea66d08f0980c75b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:22.580976 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18cad62ab6443cea66d08f0980c75b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:22.580893 18378 tablet_service.cc:1467] Processing CreateTablet for tablet c18cad62ab6443cea66d08f0980c75b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:22.580973 18509 tablet_service.cc:1467] Processing CreateTablet for tablet c18cad62ab6443cea66d08f0980c75b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:22.581352 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18cad62ab6443cea66d08f0980c75b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:22.582653 18638 tablet_service.cc:1467] Processing CreateTablet for tablet ea3ea2e3d87f4c099abbf5193c239a8b (DEFAULT_TABLE table=rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:22.583106 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3ea2e3d87f4c099abbf5193c239a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:22.583096 18249 tablet_service.cc:1467] Processing CreateTablet for tablet eeb40a038c4d404d9cb8ea693d9071d7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:22.583495 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb40a038c4d404d9cb8ea693d9071d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:22.586941 18247 tablet_service.cc:1467] Processing CreateTablet for tablet ea3ea2e3d87f4c099abbf5193c239a8b (DEFAULT_TABLE table=rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:22.587373 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3ea2e3d87f4c099abbf5193c239a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:22.587694 18639 tablet_service.cc:1467] Processing CreateTablet for tablet eeb40a038c4d404d9cb8ea693d9071d7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:22.588104 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb40a038c4d404d9cb8ea693d9071d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:22.589999 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18cad62ab6443cea66d08f0980c75b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:22.592583 18379 tablet_service.cc:1467] Processing CreateTablet for tablet eeb40a038c4d404d9cb8ea693d9071d7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:22.593050 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb40a038c4d404d9cb8ea693d9071d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:22.597659 18510 tablet_service.cc:1467] Processing CreateTablet for tablet ea3ea2e3d87f4c099abbf5193c239a8b (DEFAULT_TABLE table=rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:22.602703 19624 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:22.602834 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3ea2e3d87f4c099abbf5193c239a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:22.603622 19623 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:22.604763 19624 tablet_bootstrap.cc:654] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:22.605178 19623 tablet_bootstrap.cc:654] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:22.620105 19622 tablet_bootstrap.cc:492] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:22.624300 19622 tablet_bootstrap.cc:654] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:22.625979 19624 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:22.626060 19623 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:22.626092 19625 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:22.626294 19623 ts_tablet_manager.cc:1367] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.005s
I20240503 14:14:22.626312 19624 ts_tablet_manager.cc:1367] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.014s
I20240503 14:14:22.627090 19624 raft_consensus.cc:348] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.627281 19623 raft_consensus.cc:348] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.627444 19624 raft_consensus.cc:374] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:22.627543 19624 raft_consensus.cc:729] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.627607 19623 raft_consensus.cc:374] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:22.627812 19623 raft_consensus.cc:729] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.627889 19624 consensus_queue.cc:260] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.628131 19623 consensus_queue.cc:260] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.628603 19624 ts_tablet_manager.cc:1398] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:14:22.628974 19624 tablet_bootstrap.cc:492] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:22.630270 19625 tablet_bootstrap.cc:654] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:22.631588 19624 tablet_bootstrap.cc:654] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:22.631793 19623 ts_tablet_manager.cc:1398] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:14:22.635104 19623 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:22.637109 19623 tablet_bootstrap.cc:654] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:22.637914 19622 tablet_bootstrap.cc:492] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:22.638127 19622 ts_tablet_manager.cc:1367] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20240503 14:14:22.638978 19622 raft_consensus.cc:348] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:22.639246 19622 raft_consensus.cc:374] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:22.639410 19622 raft_consensus.cc:729] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.639503 19625 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:22.639528 19623 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:22.639680 19623 ts_tablet_manager.cc:1367] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:14:22.639748 19625 ts_tablet_manager.cc:1367] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240503 14:14:22.639735 19622 consensus_queue.cc:260] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:22.639991 19624 tablet_bootstrap.cc:492] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:22.640157 19624 ts_tablet_manager.cc:1367] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 14:14:22.640332 19622 ts_tablet_manager.cc:1398] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:14:22.640756 19622 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:22.640878 19624 raft_consensus.cc:348] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:22.641110 19624 raft_consensus.cc:374] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:22.641181 19624 raft_consensus.cc:729] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.641415 19625 raft_consensus.cc:348] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.641502 19624 consensus_queue.cc:260] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:22.641638 19625 raft_consensus.cc:374] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:22.641721 19625 raft_consensus.cc:729] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.642068 19624 ts_tablet_manager.cc:1398] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:22.642335 19624 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:22.642714 19622 tablet_bootstrap.cc:654] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:22.642632 19625 consensus_queue.cc:260] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.644398 19624 tablet_bootstrap.cc:654] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:22.647622 19625 ts_tablet_manager.cc:1398] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:14:22.648152 19625 tablet_bootstrap.cc:492] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:22.650291 19625 tablet_bootstrap.cc:654] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:22.640612 19623 raft_consensus.cc:348] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.651679 19623 raft_consensus.cc:374] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:22.651818 19623 raft_consensus.cc:729] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.656713 19623 consensus_queue.cc:260] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.658928 19623 ts_tablet_manager.cc:1398] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240503 14:14:22.659497 19625 tablet_bootstrap.cc:492] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:22.659758 19625 ts_tablet_manager.cc:1367] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240503 14:14:22.660100 19624 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:22.660172 19622 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:22.660269 19624 ts_tablet_manager.cc:1367] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.012s
I20240503 14:14:22.660352 19622 ts_tablet_manager.cc:1367] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.016s
I20240503 14:14:22.660866 19625 raft_consensus.cc:348] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:22.661031 19624 raft_consensus.cc:348] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.661079 19622 raft_consensus.cc:348] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.661250 19624 raft_consensus.cc:374] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:22.661274 19622 raft_consensus.cc:374] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:22.661203 19625 raft_consensus.cc:374] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:22.661370 19622 raft_consensus.cc:729] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.661384 19624 raft_consensus.cc:729] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.661509 19625 raft_consensus.cc:729] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.661607 19624 consensus_queue.cc:260] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.661634 19622 consensus_queue.cc:260] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.662146 19622 ts_tablet_manager.cc:1398] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:22.662173 19624 ts_tablet_manager.cc:1398] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:22.662313 19625 consensus_queue.cc:260] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:22.662859 19625 ts_tablet_manager.cc:1398] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:14:22.686215 19617 raft_consensus.cc:482] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:22.686398 19617 raft_consensus.cc:504] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.687163 19617 leader_election.cc:288] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:22.687794 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb40a038c4d404d9cb8ea693d9071d7" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:22.688046 18398 raft_consensus.cc:2454] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:22.688546 18204 leader_election.cc:302] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:22.688851 19617 raft_consensus.cc:2781] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:22.688987 19617 raft_consensus.cc:482] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:22.689081 19617 raft_consensus.cc:3037] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:22.691063 19617 raft_consensus.cc:504] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.691679 19617 leader_election.cc:288] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:22.692245 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb40a038c4d404d9cb8ea693d9071d7" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:22.692431 18398 raft_consensus.cc:3037] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:22.693575 18658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb40a038c4d404d9cb8ea693d9071d7" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:22.693717 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb40a038c4d404d9cb8ea693d9071d7" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:22.693811 18658 raft_consensus.cc:2454] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:22.694686 18398 raft_consensus.cc:2454] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:22.695030 18204 leader_election.cc:302] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters: 1c72210d694344d4b4604140088a6471
I20240503 14:14:22.695379 19617 raft_consensus.cc:2781] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:22.695971 19617 raft_consensus.cc:686] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:22.696321 19617 consensus_queue.cc:237] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.698997 18126 catalog_manager.cc:5484] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:22.858811 19617 raft_consensus.cc:482] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:22.859059 19617 raft_consensus.cc:504] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.859724 19617 leader_election.cc:288] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:22.860376 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ea2e3d87f4c099abbf5193c239a8b" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:22.860395 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ea2e3d87f4c099abbf5193c239a8b" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:22.860639 18529 raft_consensus.cc:2454] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:22.860658 18659 raft_consensus.cc:2454] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:22.861104 18202 leader_election.cc:302] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:22.861439 19617 raft_consensus.cc:2781] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:22.861531 19617 raft_consensus.cc:482] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:22.861647 19617 raft_consensus.cc:3037] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:22.863492 19617 raft_consensus.cc:504] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.864084 19617 leader_election.cc:288] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:22.864643 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ea2e3d87f4c099abbf5193c239a8b" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:22.864665 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ea2e3d87f4c099abbf5193c239a8b" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:22.864862 18529 raft_consensus.cc:3037] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:22.864882 18659 raft_consensus.cc:3037] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:22.867710 18529 raft_consensus.cc:2454] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:22.867698 18659 raft_consensus.cc:2454] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:22.868263 18202 leader_election.cc:302] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:22.868636 19617 raft_consensus.cc:2781] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:22.868831 19617 raft_consensus.cc:686] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:22.869159 19617 consensus_queue.cc:237] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:22.873183 18126 catalog_manager.cc:5484] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:22.920471 19617 raft_consensus.cc:482] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:22.920727 19617 raft_consensus.cc:504] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:22.921607 19617 leader_election.cc:288] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:22.922067 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18cad62ab6443cea66d08f0980c75b6" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:22.922156 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18cad62ab6443cea66d08f0980c75b6" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:22.922348 18529 raft_consensus.cc:2454] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:22.922399 18398 raft_consensus.cc:2454] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:22.922832 18205 leader_election.cc:302] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:22.923099 19617 raft_consensus.cc:2781] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:22.923213 19617 raft_consensus.cc:482] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:22.923293 19617 raft_consensus.cc:3037] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:22.925307 19617 raft_consensus.cc:504] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:22.925969 19617 leader_election.cc:288] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:22.926496 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18cad62ab6443cea66d08f0980c75b6" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:22.926491 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18cad62ab6443cea66d08f0980c75b6" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:22.926678 18529 raft_consensus.cc:3037] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:22.926757 18398 raft_consensus.cc:3037] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:22.928534 18529 raft_consensus.cc:2454] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:22.929010 18205 leader_election.cc:302] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:22.929327 19617 raft_consensus.cc:2781] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:22.929461 19617 raft_consensus.cc:686] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:22.929580 18398 raft_consensus.cc:2454] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:22.929769 19617 consensus_queue.cc:237] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:22.932879 18126 catalog_manager.cc:5484] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:23.155732 18659 raft_consensus.cc:3037] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:23.158541 19617 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:23.167091 19637 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:23.332026 19637 consensus_queue.cc:1035] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:23.336366 19641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:23.336817 19641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:14:23.336719 19617 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:23.337599 19617 consensus_queue.cc:1035] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:23.347445 19617 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:23.348613 19641 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 230.104 ms
time per row: 0.115052 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:14:24.594225 18126 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49596:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:24.604022 18509 tablet_service.cc:1467] Processing CreateTablet for tablet f09411ea6c414451b66fd457d9aaf010 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:24.603928 18379 tablet_service.cc:1467] Processing CreateTablet for tablet f09411ea6c414451b66fd457d9aaf010 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:24.604346 18638 tablet_service.cc:1467] Processing CreateTablet for tablet f09411ea6c414451b66fd457d9aaf010 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:24.604509 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09411ea6c414451b66fd457d9aaf010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:24.604516 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09411ea6c414451b66fd457d9aaf010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:24.604738 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09411ea6c414451b66fd457d9aaf010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:24.606777 18508 tablet_service.cc:1467] Processing CreateTablet for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e (DEFAULT_TABLE table=rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:24.607203 18508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:24.609675 18510 tablet_service.cc:1467] Processing CreateTablet for tablet 9577e6d3801f42f89afe8a48a77031dd (DEFAULT_TABLE table=rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:24.610081 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9577e6d3801f42f89afe8a48a77031dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:24.610394 18639 tablet_service.cc:1467] Processing CreateTablet for tablet 9577e6d3801f42f89afe8a48a77031dd (DEFAULT_TABLE table=rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:24.610760 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9577e6d3801f42f89afe8a48a77031dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:24.613781 18380 tablet_service.cc:1467] Processing CreateTablet for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e (DEFAULT_TABLE table=rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:24.614204 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:24.617657 19702 tablet_bootstrap.cc:492] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:24.617929 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 9577e6d3801f42f89afe8a48a77031dd (DEFAULT_TABLE table=rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:24.618389 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9577e6d3801f42f89afe8a48a77031dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:24.618955 19702 tablet_bootstrap.cc:654] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:24.620522 19703 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:24.620810 19702 tablet_bootstrap.cc:492] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:24.620946 19702 ts_tablet_manager.cc:1367] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:14:24.621728 18249 tablet_service.cc:1467] Processing CreateTablet for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e (DEFAULT_TABLE table=rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:24.621773 19702 raft_consensus.cc:348] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.621976 19702 raft_consensus.cc:374] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:24.622059 19702 raft_consensus.cc:729] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:24.622179 19703 tablet_bootstrap.cc:654] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:24.622215 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:24.622329 19702 consensus_queue.cc:260] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.623031 19702 ts_tablet_manager.cc:1398] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:24.623317 19702 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:24.625010 19702 tablet_bootstrap.cc:654] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:24.635063 19703 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:24.635282 19703 ts_tablet_manager.cc:1367] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20240503 14:14:24.636140 19703 raft_consensus.cc:348] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.636428 19703 raft_consensus.cc:374] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:24.636533 19703 raft_consensus.cc:729] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:24.636785 19703 consensus_queue.cc:260] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.640692 19703 ts_tablet_manager.cc:1398] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:14:24.642251 19703 tablet_bootstrap.cc:492] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:24.644255 19703 tablet_bootstrap.cc:654] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:24.667299 19709 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:24.668115 19706 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:24.669903 19706 tablet_bootstrap.cc:654] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:24.670904 19709 tablet_bootstrap.cc:654] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:24.672397 19702 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:24.672709 19702 ts_tablet_manager.cc:1367] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.049s user 0.020s sys 0.027s
I20240503 14:14:24.673856 19702 raft_consensus.cc:348] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:24.674340 19702 raft_consensus.cc:374] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:24.674403 19703 tablet_bootstrap.cc:492] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:24.674481 19702 raft_consensus.cc:729] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:24.674681 19703 ts_tablet_manager.cc:1367] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.014s
I20240503 14:14:24.674963 19702 consensus_queue.cc:260] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:24.675572 19703 raft_consensus.cc:348] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.675766 19706 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:24.675864 19703 raft_consensus.cc:374] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:24.675957 19703 raft_consensus.cc:729] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:24.675971 19706 ts_tablet_manager.cc:1367] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240503 14:14:24.676288 19703 consensus_queue.cc:260] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.676919 19702 ts_tablet_manager.cc:1398] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:14:24.676872 19706 raft_consensus.cc:348] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.677160 19706 raft_consensus.cc:374] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:24.677357 19702 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:24.679136 19703 ts_tablet_manager.cc:1398] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:14:24.680971 19702 tablet_bootstrap.cc:654] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:24.683032 19706 raft_consensus.cc:729] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:24.684118 19702 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:24.684412 19702 ts_tablet_manager.cc:1367] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240503 14:14:24.685361 19702 raft_consensus.cc:348] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.685631 19702 raft_consensus.cc:374] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:24.685750 19702 raft_consensus.cc:729] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:24.685547 19706 consensus_queue.cc:260] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.686018 19702 consensus_queue.cc:260] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.686264 19706 ts_tablet_manager.cc:1398] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 14:14:24.687316 19706 tablet_bootstrap.cc:492] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:24.687395 19702 ts_tablet_manager.cc:1398] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:24.689041 19706 tablet_bootstrap.cc:654] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:24.689713 19709 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:24.689921 19709 ts_tablet_manager.cc:1367] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.009s
I20240503 14:14:24.690773 19709 raft_consensus.cc:348] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:24.691035 19709 raft_consensus.cc:374] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:24.691148 19709 raft_consensus.cc:729] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:24.691406 19709 consensus_queue.cc:260] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:24.692163 19709 ts_tablet_manager.cc:1398] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:24.693653 19706 tablet_bootstrap.cc:492] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:24.693815 19706 ts_tablet_manager.cc:1367] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240503 14:14:24.694504 19706 raft_consensus.cc:348] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.694680 19706 raft_consensus.cc:374] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:24.694751 19706 raft_consensus.cc:729] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:24.694991 19706 consensus_queue.cc:260] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.695461 19706 ts_tablet_manager.cc:1398] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:24.695796 19706 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:24.697120 19706 tablet_bootstrap.cc:654] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:24.698834 19706 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:24.698971 19706 ts_tablet_manager.cc:1367] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:14:24.699568 19706 raft_consensus.cc:348] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:24.699736 19706 raft_consensus.cc:374] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:24.699824 19706 raft_consensus.cc:729] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:24.700035 19706 consensus_queue.cc:260] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:24.700462 19706 ts_tablet_manager.cc:1398] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:14:24.725023 19617 raft_consensus.cc:482] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:24.725307 19617 raft_consensus.cc:504] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:24.726017 19617 leader_election.cc:288] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:24.726600 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39fbf1d8f6f46ccab3c2b992ed0c63e" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:24.726625 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39fbf1d8f6f46ccab3c2b992ed0c63e" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:24.726852 18530 raft_consensus.cc:2454] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:24.726878 18399 raft_consensus.cc:2454] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:24.727382 18204 leader_election.cc:302] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:24.727708 19617 raft_consensus.cc:2781] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:24.727811 19617 raft_consensus.cc:482] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:24.727893 19617 raft_consensus.cc:3037] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:24.729794 19617 raft_consensus.cc:504] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:24.730456 19617 leader_election.cc:288] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:24.730978 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39fbf1d8f6f46ccab3c2b992ed0c63e" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:24.730999 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39fbf1d8f6f46ccab3c2b992ed0c63e" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:24.731200 18530 raft_consensus.cc:3037] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:24.731213 18399 raft_consensus.cc:3037] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:24.733151 18530 raft_consensus.cc:2454] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:24.733171 18399 raft_consensus.cc:2454] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:24.733669 18204 leader_election.cc:302] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:24.733980 19617 raft_consensus.cc:2781] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:24.734405 19617 raft_consensus.cc:686] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:24.734651 19617 consensus_queue.cc:237] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:24.737710 18126 catalog_manager.cc:5484] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:24.777837 19600 raft_consensus.cc:482] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:24.778086 19600 raft_consensus.cc:504] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.778903 19600 leader_election.cc:288] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:24.779582 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09411ea6c414451b66fd457d9aaf010" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:24.779620 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09411ea6c414451b66fd457d9aaf010" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:24.779871 18399 raft_consensus.cc:2454] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:24.779851 18530 raft_consensus.cc:2454] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:24.780368 18596 leader_election.cc:302] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:24.780686 19600 raft_consensus.cc:2781] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:24.780790 19600 raft_consensus.cc:482] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:24.780854 19600 raft_consensus.cc:3037] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:24.782895 19600 raft_consensus.cc:504] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.783572 19600 leader_election.cc:288] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:24.784211 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09411ea6c414451b66fd457d9aaf010" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:24.784296 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09411ea6c414451b66fd457d9aaf010" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:24.784475 18530 raft_consensus.cc:3037] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:24.784471 18399 raft_consensus.cc:3037] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:24.787353 18530 raft_consensus.cc:2454] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:24.787341 18399 raft_consensus.cc:2454] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:24.787905 18595 leader_election.cc:302] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:24.788306 19600 raft_consensus.cc:2781] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:24.788666 19600 raft_consensus.cc:686] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Becoming Leader. State: Replica: 1c72210d694344d4b4604140088a6471, State: Running, Role: LEADER
I20240503 14:14:24.789005 19600 consensus_queue.cc:237] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.791960 18126 catalog_manager.cc:5484] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c72210d694344d4b4604140088a6471 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:24.802664 19600 raft_consensus.cc:482] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:24.802893 19600 raft_consensus.cc:504] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.803520 19600 leader_election.cc:288] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:24.804131 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9577e6d3801f42f89afe8a48a77031dd" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:24.804100 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9577e6d3801f42f89afe8a48a77031dd" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:24.804368 18399 raft_consensus.cc:2454] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:24.804415 18530 raft_consensus.cc:2454] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:24.804821 18596 leader_election.cc:302] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:24.805114 19600 raft_consensus.cc:2781] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:24.805199 19600 raft_consensus.cc:482] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:24.805316 19600 raft_consensus.cc:3037] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:24.807204 19600 raft_consensus.cc:504] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.807906 19600 leader_election.cc:288] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Requested vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:24.808310 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9577e6d3801f42f89afe8a48a77031dd" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:24.808557 18530 raft_consensus.cc:3037] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:24.808559 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9577e6d3801f42f89afe8a48a77031dd" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:24.808720 18399 raft_consensus.cc:3037] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:24.810506 18530 raft_consensus.cc:2454] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:24.810659 18399 raft_consensus.cc:2454] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:24.810930 18596 leader_election.cc:302] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:24.811214 19600 raft_consensus.cc:2781] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:24.811379 19600 raft_consensus.cc:686] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Becoming Leader. State: Replica: 1c72210d694344d4b4604140088a6471, State: Running, Role: LEADER
I20240503 14:14:24.811810 19600 consensus_queue.cc:237] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:24.814801 18126 catalog_manager.cc:5484] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c72210d694344d4b4604140088a6471 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:25.180941 19600 consensus_queue.cc:1035] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:25.187515 19600 consensus_queue.cc:1035] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:25.219373 19720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:25.219728 19720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:25.227645 19720 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 14:14:25.272835 19617 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:25.311914 19617 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:25.345584 19600 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:25.353471 19717 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 294.773 ms
time per row: 0.147386 ms
I20240503 14:14:25.672358 18124 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49596:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:25.681870 18639 tablet_service.cc:1467] Processing CreateTablet for tablet 79373bd79a934eaab67bbcf2208e7b5b (DEFAULT_TABLE table=rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:25.681870 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 79373bd79a934eaab67bbcf2208e7b5b (DEFAULT_TABLE table=rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:25.682400 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79373bd79a934eaab67bbcf2208e7b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:25.682399 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79373bd79a934eaab67bbcf2208e7b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:25.683224 18248 tablet_service.cc:1467] Processing CreateTablet for tablet 91743575cf4a4f38b5b43265539237a3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:25.683656 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91743575cf4a4f38b5b43265539237a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:25.686054 18247 tablet_service.cc:1467] Processing CreateTablet for tablet 10a4ca83469d4dab823a26d6cbe952a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:25.686285 18249 tablet_service.cc:1467] Processing CreateTablet for tablet 79373bd79a934eaab67bbcf2208e7b5b (DEFAULT_TABLE table=rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:25.686502 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a4ca83469d4dab823a26d6cbe952a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:25.686779 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79373bd79a934eaab67bbcf2208e7b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:25.687006 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 91743575cf4a4f38b5b43265539237a3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:25.687417 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91743575cf4a4f38b5b43265539237a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:25.687700 18510 tablet_service.cc:1467] Processing CreateTablet for tablet 10a4ca83469d4dab823a26d6cbe952a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:25.688104 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a4ca83469d4dab823a26d6cbe952a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:25.694710 18638 tablet_service.cc:1467] Processing CreateTablet for tablet 10a4ca83469d4dab823a26d6cbe952a6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:25.695209 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a4ca83469d4dab823a26d6cbe952a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:25.697767 18640 tablet_service.cc:1467] Processing CreateTablet for tablet 91743575cf4a4f38b5b43265539237a3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:25.698240 18640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91743575cf4a4f38b5b43265539237a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:25.702903 19789 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:25.705341 19789 tablet_bootstrap.cc:654] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:25.706830 19791 tablet_bootstrap.cc:492] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:25.709259 19791 tablet_bootstrap.cc:654] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:25.714937 19788 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:25.716526 19789 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:25.716838 19789 ts_tablet_manager.cc:1367] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240503 14:14:25.717914 19789 raft_consensus.cc:348] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:25.718402 19789 raft_consensus.cc:374] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:25.718595 19789 raft_consensus.cc:729] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:25.718972 19789 consensus_queue.cc:260] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:25.719329 19788 tablet_bootstrap.cc:654] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:25.720001 19789 ts_tablet_manager.cc:1398] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:25.720431 19789 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:25.722349 19789 tablet_bootstrap.cc:654] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:25.731079 19789 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:25.731433 19789 ts_tablet_manager.cc:1367] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240503 14:14:25.732399 19789 raft_consensus.cc:348] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:25.732807 19790 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:25.732810 19789 raft_consensus.cc:374] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:25.732980 19789 raft_consensus.cc:729] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:25.733424 19789 consensus_queue.cc:260] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:25.735101 19789 ts_tablet_manager.cc:1398] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:14:25.738312 19790 tablet_bootstrap.cc:654] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:25.738555 19788 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:25.738808 19788 ts_tablet_manager.cc:1367] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.008s
I20240503 14:14:25.739027 19791 tablet_bootstrap.cc:492] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:25.739225 19791 ts_tablet_manager.cc:1367] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.022s
I20240503 14:14:25.739792 19788 raft_consensus.cc:348] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:25.740094 19788 raft_consensus.cc:374] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:25.740017 19791 raft_consensus.cc:348] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:25.740242 19791 raft_consensus.cc:374] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:25.740272 19788 raft_consensus.cc:729] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:25.740341 19791 raft_consensus.cc:729] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:25.740522 19788 consensus_queue.cc:260] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:25.740592 19791 consensus_queue.cc:260] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:25.741156 19791 ts_tablet_manager.cc:1398] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:14:25.741390 19788 ts_tablet_manager.cc:1398] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:25.741689 19788 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:25.745149 19788 tablet_bootstrap.cc:654] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:25.747053 19788 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:25.747251 19788 ts_tablet_manager.cc:1367] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:14:25.747962 19788 raft_consensus.cc:348] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:25.748204 19788 raft_consensus.cc:374] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:25.748291 19788 raft_consensus.cc:729] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:25.748534 19788 consensus_queue.cc:260] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:25.749049 19788 ts_tablet_manager.cc:1398] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:25.749451 19788 tablet_bootstrap.cc:492] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:25.750038 19790 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:25.750200 19790 ts_tablet_manager.cc:1367] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20240503 14:14:25.750838 19790 raft_consensus.cc:348] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:25.751009 19790 raft_consensus.cc:374] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:25.751119 19790 raft_consensus.cc:729] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:25.751173 19788 tablet_bootstrap.cc:654] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:25.751361 19790 consensus_queue.cc:260] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:25.751889 19790 ts_tablet_manager.cc:1398] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:25.752200 19790 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:25.754756 19790 tablet_bootstrap.cc:654] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:25.756778 19790 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:25.756947 19790 ts_tablet_manager.cc:1367] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:14:25.757740 19790 raft_consensus.cc:348] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:25.758018 19790 raft_consensus.cc:374] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:25.758117 19790 raft_consensus.cc:729] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:25.758538 19788 tablet_bootstrap.cc:492] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:25.758447 19790 consensus_queue.cc:260] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:25.758690 19788 ts_tablet_manager.cc:1367] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.006s
I20240503 14:14:25.759043 19790 ts_tablet_manager.cc:1398] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:25.759352 19790 tablet_bootstrap.cc:492] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:25.759470 19788 raft_consensus.cc:348] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:25.759650 19788 raft_consensus.cc:374] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:25.759740 19788 raft_consensus.cc:729] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:25.760031 19617 raft_consensus.cc:482] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:25.759977 19788 consensus_queue.cc:260] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:25.760188 19617 raft_consensus.cc:504] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:25.760691 19788 ts_tablet_manager.cc:1398] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:25.760912 19617 leader_election.cc:288] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:25.760975 19790 tablet_bootstrap.cc:654] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:25.761662 18658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79373bd79a934eaab67bbcf2208e7b5b" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:25.761907 18658 raft_consensus.cc:2454] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:25.762243 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79373bd79a934eaab67bbcf2208e7b5b" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:25.762408 18202 leader_election.cc:302] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:25.762493 18399 raft_consensus.cc:2454] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:25.763108 19790 tablet_bootstrap.cc:492] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:25.763275 19790 ts_tablet_manager.cc:1367] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 14:14:25.764439 19790 raft_consensus.cc:348] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:25.764660 19790 raft_consensus.cc:374] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:25.764758 19790 raft_consensus.cc:729] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:25.764990 19790 consensus_queue.cc:260] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:25.765619 19617 raft_consensus.cc:2781] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:25.765720 19790 ts_tablet_manager.cc:1398] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:25.765748 19617 raft_consensus.cc:482] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:25.765837 19617 raft_consensus.cc:3037] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:25.767681 19617 raft_consensus.cc:504] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:25.768314 19617 leader_election.cc:288] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:25.768780 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79373bd79a934eaab67bbcf2208e7b5b" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:25.769001 18659 raft_consensus.cc:3037] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:25.769790 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79373bd79a934eaab67bbcf2208e7b5b" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:25.770016 18399 raft_consensus.cc:3037] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:25.771627 18659 raft_consensus.cc:2454] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:25.772064 18202 leader_election.cc:302] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:25.772375 19617 raft_consensus.cc:2781] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:25.772437 18399 raft_consensus.cc:2454] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:25.772711 19617 raft_consensus.cc:686] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:25.773061 19617 consensus_queue.cc:237] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:25.776074 18123 catalog_manager.cc:5484] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:25.811324 19617 raft_consensus.cc:482] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:25.811558 19617 raft_consensus.cc:504] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:25.812207 19617 leader_election.cc:288] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:25.812657 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4ca83469d4dab823a26d6cbe952a6" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:25.812887 18659 raft_consensus.cc:2454] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:25.812829 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4ca83469d4dab823a26d6cbe952a6" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:25.813050 18530 raft_consensus.cc:2454] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:25.813318 18202 leader_election.cc:302] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:25.813645 19617 raft_consensus.cc:2781] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:25.813766 19617 raft_consensus.cc:482] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:25.813869 19617 raft_consensus.cc:3037] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:25.815977 19617 raft_consensus.cc:504] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:25.816681 19617 leader_election.cc:288] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:25.817142 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4ca83469d4dab823a26d6cbe952a6" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:25.817216 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a4ca83469d4dab823a26d6cbe952a6" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:25.817364 18659 raft_consensus.cc:3037] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:25.817469 18530 raft_consensus.cc:3037] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:25.819381 18659 raft_consensus.cc:2454] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:25.819380 18530 raft_consensus.cc:2454] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:25.819908 18202 leader_election.cc:302] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:25.820236 19617 raft_consensus.cc:2781] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:25.820405 19617 raft_consensus.cc:686] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:25.820657 19617 consensus_queue.cc:237] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:25.823760 18123 catalog_manager.cc:5484] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:25.891806 19617 raft_consensus.cc:482] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:25.892051 19617 raft_consensus.cc:504] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:25.892786 19617 leader_election.cc:288] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:25.893397 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91743575cf4a4f38b5b43265539237a3" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:25.893429 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91743575cf4a4f38b5b43265539237a3" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:25.893647 18659 raft_consensus.cc:2454] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:25.893676 18399 raft_consensus.cc:2454] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:25.894155 18204 leader_election.cc:302] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:25.894466 19617 raft_consensus.cc:2781] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:25.894577 19617 raft_consensus.cc:482] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:25.894654 19617 raft_consensus.cc:3037] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:25.896613 19617 raft_consensus.cc:504] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:25.897302 19617 leader_election.cc:288] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:25.897969 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91743575cf4a4f38b5b43265539237a3" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:25.897964 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91743575cf4a4f38b5b43265539237a3" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:25.898172 18399 raft_consensus.cc:3037] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:25.898211 18659 raft_consensus.cc:3037] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:25.900324 18399 raft_consensus.cc:2454] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:25.900318 18659 raft_consensus.cc:2454] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:25.900883 18202 leader_election.cc:302] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:25.901233 19617 raft_consensus.cc:2781] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:25.901428 19617 raft_consensus.cc:686] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:25.901839 19617 consensus_queue.cc:237] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:25.905019 18123 catalog_manager.cc:5484] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:26.253602 19617 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:26.260558 19772 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:26.265075 19772 consensus_queue.cc:1035] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:26.272735 19617 consensus_queue.cc:1035] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:26.345139 19804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:26.345587 19804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:26.354327 19804 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:14:26.364157 19807 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:26.372591 19617 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 221.493 ms
time per row: 0.110746 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:14:27.611266 18124 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49596:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:27.621474 18510 tablet_service.cc:1467] Processing CreateTablet for tablet 87fded61bfc240f5846080c328528edc (DEFAULT_TABLE table=rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:27.621548 18249 tablet_service.cc:1467] Processing CreateTablet for tablet 87fded61bfc240f5846080c328528edc (DEFAULT_TABLE table=rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:27.621970 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fded61bfc240f5846080c328528edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:27.621976 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fded61bfc240f5846080c328528edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:27.622705 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 87fded61bfc240f5846080c328528edc (DEFAULT_TABLE table=rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:27.623129 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fded61bfc240f5846080c328528edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:27.623042 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 71a326db29114366951a67b9aa84ac03 (DEFAULT_TABLE table=rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:27.623394 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a326db29114366951a67b9aa84ac03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:27.626993 18248 tablet_service.cc:1467] Processing CreateTablet for tablet c35376539bc944e59b74bfe59d5feb9e (DEFAULT_TABLE table=rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:27.626993 18509 tablet_service.cc:1467] Processing CreateTablet for tablet c35376539bc944e59b74bfe59d5feb9e (DEFAULT_TABLE table=rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:27.627430 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35376539bc944e59b74bfe59d5feb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:27.627430 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35376539bc944e59b74bfe59d5feb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:27.629751 18508 tablet_service.cc:1467] Processing CreateTablet for tablet 71a326db29114366951a67b9aa84ac03 (DEFAULT_TABLE table=rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:27.630215 18508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a326db29114366951a67b9aa84ac03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:27.630278 18638 tablet_service.cc:1467] Processing CreateTablet for tablet 71a326db29114366951a67b9aa84ac03 (DEFAULT_TABLE table=rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:27.630692 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a326db29114366951a67b9aa84ac03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:27.632833 18639 tablet_service.cc:1467] Processing CreateTablet for tablet c35376539bc944e59b74bfe59d5feb9e (DEFAULT_TABLE table=rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:27.633430 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35376539bc944e59b74bfe59d5feb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:27.645934 19873 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:27.648845 19873 tablet_bootstrap.cc:654] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:27.656347 19872 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:27.657344 19874 tablet_bootstrap.cc:492] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:27.659612 19872 tablet_bootstrap.cc:654] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:27.661795 19874 tablet_bootstrap.cc:654] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:27.665917 19872 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:27.666191 19872 ts_tablet_manager.cc:1367] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20240503 14:14:27.666222 19871 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:27.667063 19872 raft_consensus.cc:348] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:27.667311 19872 raft_consensus.cc:374] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:27.667435 19872 raft_consensus.cc:729] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:27.667491 19873 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:27.667652 19873 ts_tablet_manager.cc:1367] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20240503 14:14:27.667690 19872 consensus_queue.cc:260] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:27.667951 19871 tablet_bootstrap.cc:654] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:27.668301 19872 ts_tablet_manager.cc:1398] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:14:27.668366 19873 raft_consensus.cc:348] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:27.668556 19873 raft_consensus.cc:374] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:27.668671 19873 raft_consensus.cc:729] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:27.668723 19872 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:27.668906 19873 consensus_queue.cc:260] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:27.669479 19873 ts_tablet_manager.cc:1398] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:27.669569 19874 tablet_bootstrap.cc:492] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:27.669735 19874 ts_tablet_manager.cc:1367] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 14:14:27.669754 19873 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:27.670552 19874 raft_consensus.cc:348] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:27.670802 19874 raft_consensus.cc:374] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:27.670883 19874 raft_consensus.cc:729] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:27.671134 19874 consensus_queue.cc:260] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:27.671348 19873 tablet_bootstrap.cc:654] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:27.671722 19874 ts_tablet_manager.cc:1398] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:27.672961 19874 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:27.674801 19874 tablet_bootstrap.cc:654] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:27.675679 19369 raft_consensus.cc:482] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:27.675873 19369 raft_consensus.cc:504] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:27.676667 19369 leader_election.cc:288] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:27.677336 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a326db29114366951a67b9aa84ac03" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:27.677495 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a326db29114366951a67b9aa84ac03" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:27.677592 18659 raft_consensus.cc:2454] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 0.
I20240503 14:14:27.678356 18463 leader_election.cc:302] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
W20240503 14:14:27.678710 18465 leader_election.cc:341] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:14:27.680371 19369 raft_consensus.cc:2781] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:27.680503 19369 raft_consensus.cc:482] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:27.680611 19369 raft_consensus.cc:3037] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:27.681128 19874 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:27.681332 19874 ts_tablet_manager.cc:1367] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:14:27.682128 19874 raft_consensus.cc:348] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:27.682353 19874 raft_consensus.cc:374] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:27.682430 19874 raft_consensus.cc:729] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:27.682689 19874 consensus_queue.cc:260] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:27.683413 19369 raft_consensus.cc:504] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:27.684005 19874 ts_tablet_manager.cc:1398] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:14:27.684922 19871 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:27.685097 19871 ts_tablet_manager.cc:1367] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20240503 14:14:27.685985 19871 raft_consensus.cc:348] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:27.686213 19871 raft_consensus.cc:374] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:27.686308 19871 raft_consensus.cc:729] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:27.686556 19871 consensus_queue.cc:260] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:27.687120 19871 ts_tablet_manager.cc:1398] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:14:27.687409 19871 tablet_bootstrap.cc:492] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:27.689167 19871 tablet_bootstrap.cc:654] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:27.689910 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a326db29114366951a67b9aa84ac03" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:27.690162 18659 raft_consensus.cc:3037] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:27.693181 18659 raft_consensus.cc:2454] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 1.
I20240503 14:14:27.694460 18463 leader_election.cc:302] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:27.698763 19369 leader_election.cc:288] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:27.699927 19369 raft_consensus.cc:2781] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:27.701092 19369 raft_consensus.cc:686] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Becoming Leader. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Running, Role: LEADER
I20240503 14:14:27.701753 19369 consensus_queue.cc:237] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:27.707875 19809 raft_consensus.cc:482] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:27.708115 19809 raft_consensus.cc:504] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:27.709061 19809 leader_election.cc:288] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:27.711146 19872 tablet_bootstrap.cc:654] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:27.711621 18124 catalog_manager.cc:5484] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3). New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:27.711186 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a326db29114366951a67b9aa84ac03" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:27.712761 18659 raft_consensus.cc:2379] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea366e6e15a24c52b1ea5a154f942568 in current term 1: Already voted for candidate 55339e38e7aa4bb7b7b42aa206108722 in this term.
I20240503 14:14:27.720664 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a326db29114366951a67b9aa84ac03" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:27.720685 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a326db29114366951a67b9aa84ac03" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:27.721053 18398 raft_consensus.cc:3037] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:27.721652 18336 leader_election.cc:302] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea366e6e15a24c52b1ea5a154f942568; no voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722
I20240503 14:14:27.726159 18398 raft_consensus.cc:2454] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 1.
I20240503 14:14:27.726878 19809 raft_consensus.cc:2726] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:14:27.728546 19872 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:27.729144 19872 ts_tablet_manager.cc:1367] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.060s user 0.000s sys 0.005s
I20240503 14:14:27.730355 19872 raft_consensus.cc:348] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:27.730659 19872 raft_consensus.cc:374] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:27.730772 19872 raft_consensus.cc:729] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:27.731073 19872 consensus_queue.cc:260] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:27.732978 19872 ts_tablet_manager.cc:1398] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:14:27.733708 19871 tablet_bootstrap.cc:492] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:27.733915 19871 ts_tablet_manager.cc:1367] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.047s user 0.030s sys 0.013s
I20240503 14:14:27.734452 19873 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:27.734652 19873 ts_tablet_manager.cc:1367] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.065s user 0.028s sys 0.032s
I20240503 14:14:27.735354 19873 raft_consensus.cc:348] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:27.734612 19871 raft_consensus.cc:348] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:27.735551 19873 raft_consensus.cc:374] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:27.735608 19871 raft_consensus.cc:374] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:27.735656 19873 raft_consensus.cc:729] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:27.735713 19871 raft_consensus.cc:729] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:27.735944 19873 consensus_queue.cc:260] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:27.736028 19871 consensus_queue.cc:260] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:27.736500 19873 ts_tablet_manager.cc:1398] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:27.736910 19873 tablet_bootstrap.cc:492] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:27.736616 19871 ts_tablet_manager.cc:1398] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:27.738792 19873 tablet_bootstrap.cc:654] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:27.742058 19873 tablet_bootstrap.cc:492] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:27.742301 19873 ts_tablet_manager.cc:1367] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:14:27.742998 19873 raft_consensus.cc:348] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:27.743175 19873 raft_consensus.cc:374] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:27.743242 19873 raft_consensus.cc:729] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:27.743419 19873 consensus_queue.cc:260] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:27.743832 19873 ts_tablet_manager.cc:1398] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:14:27.837072 19881 raft_consensus.cc:482] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:27.837350 19881 raft_consensus.cc:504] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:27.838073 19881 leader_election.cc:288] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:27.838891 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35376539bc944e59b74bfe59d5feb9e" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:27.838866 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35376539bc944e59b74bfe59d5feb9e" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:27.839151 18268 raft_consensus.cc:2454] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 0.
I20240503 14:14:27.839172 18659 raft_consensus.cc:2454] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 0.
I20240503 14:14:27.839674 18463 leader_election.cc:302] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:27.839957 19881 raft_consensus.cc:2781] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:27.840095 19881 raft_consensus.cc:482] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:27.840183 19881 raft_consensus.cc:3037] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:27.842190 19881 raft_consensus.cc:504] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:27.842813 19881 leader_election.cc:288] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:27.843444 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35376539bc944e59b74bfe59d5feb9e" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:27.843461 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35376539bc944e59b74bfe59d5feb9e" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:27.843674 18268 raft_consensus.cc:3037] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:27.843693 18659 raft_consensus.cc:3037] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:27.846611 18268 raft_consensus.cc:2454] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 1.
I20240503 14:14:27.846596 18659 raft_consensus.cc:2454] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 1.
I20240503 14:14:27.847229 18463 leader_election.cc:302] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:27.847661 19881 raft_consensus.cc:2781] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:27.847808 19881 raft_consensus.cc:686] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Becoming Leader. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Running, Role: LEADER
I20240503 14:14:27.848120 19881 consensus_queue.cc:237] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:27.851315 18127 catalog_manager.cc:5484] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3). New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:27.931394 19886 raft_consensus.cc:482] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:27.931587 19886 raft_consensus.cc:504] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:27.932312 19886 leader_election.cc:288] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:27.932899 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fded61bfc240f5846080c328528edc" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:27.932973 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fded61bfc240f5846080c328528edc" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:27.933126 18529 raft_consensus.cc:2454] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:27.933243 18268 raft_consensus.cc:2454] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:27.933552 18336 leader_election.cc:302] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:27.933840 19886 raft_consensus.cc:2781] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:27.933964 19886 raft_consensus.cc:482] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:27.934062 19886 raft_consensus.cc:3037] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:27.936035 19886 raft_consensus.cc:504] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:27.936726 19886 leader_election.cc:288] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Requested vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:27.937256 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fded61bfc240f5846080c328528edc" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:27.937366 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fded61bfc240f5846080c328528edc" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:27.937546 18529 raft_consensus.cc:3037] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:27.937631 18268 raft_consensus.cc:3037] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:27.940459 18529 raft_consensus.cc:2454] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:27.940459 18268 raft_consensus.cc:2454] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:27.941000 18336 leader_election.cc:302] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:27.941342 19886 raft_consensus.cc:2781] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:27.941531 19886 raft_consensus.cc:686] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Becoming Leader. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Running, Role: LEADER
I20240503 14:14:27.941924 19886 consensus_queue.cc:237] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } }
I20240503 14:14:27.945091 18127 catalog_manager.cc:5484] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:28.246461 19881 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:28.267763 19883 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:28.333771 19891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:28.334115 19891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:28.341936 19891 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:14:28.376152 19883 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240503 14:14:28.383853 19883 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:28.495260 19886 consensus_queue.cc:1035] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:28.528182 18267 raft_consensus.cc:1261] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:28.531550 19934 consensus_queue.cc:1035] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 281.778 ms
time per row: 0.140889 ms
I20240503 14:14:28.795418 18122 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49596:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:28.806025 18249 tablet_service.cc:1467] Processing CreateTablet for tablet f2c5c6f4317649fab75d01e339b76403 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:28.806495 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c5c6f4317649fab75d01e339b76403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:28.806403 18378 tablet_service.cc:1467] Processing CreateTablet for tablet f2c5c6f4317649fab75d01e339b76403 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:28.806840 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c5c6f4317649fab75d01e339b76403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:28.807857 18510 tablet_service.cc:1467] Processing CreateTablet for tablet f2c5c6f4317649fab75d01e339b76403 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:28.807929 18509 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb29bdb71f24b46b32e7b97d0172c20 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:28.808341 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c5c6f4317649fab75d01e339b76403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:28.808615 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb29bdb71f24b46b32e7b97d0172c20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:28.810649 18638 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb29bdb71f24b46b32e7b97d0172c20 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:28.811101 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb29bdb71f24b46b32e7b97d0172c20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:28.811370 18248 tablet_service.cc:1467] Processing CreateTablet for tablet c2167226099a4dcf98304918e8f412af (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:28.811776 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2167226099a4dcf98304918e8f412af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:28.814117 18247 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb29bdb71f24b46b32e7b97d0172c20 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:28.814584 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb29bdb71f24b46b32e7b97d0172c20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:28.817814 19961 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:28.819494 18639 tablet_service.cc:1467] Processing CreateTablet for tablet c2167226099a4dcf98304918e8f412af (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:28.819733 19961 tablet_bootstrap.cc:654] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:28.820113 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2167226099a4dcf98304918e8f412af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:28.827329 19962 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:28.827651 18380 tablet_service.cc:1467] Processing CreateTablet for tablet c2167226099a4dcf98304918e8f412af (DEFAULT_TABLE table=rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:28.828226 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2167226099a4dcf98304918e8f412af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:28.829519 19962 tablet_bootstrap.cc:654] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:28.829922 19961 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:28.830199 19961 ts_tablet_manager.cc:1367] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20240503 14:14:28.831336 19961 raft_consensus.cc:348] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.831797 19961 raft_consensus.cc:374] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:28.831985 19961 raft_consensus.cc:729] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:28.832517 19961 consensus_queue.cc:260] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.834750 19961 ts_tablet_manager.cc:1398] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:14:28.839393 19962 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:28.839691 19962 ts_tablet_manager.cc:1367] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 14:14:28.841060 19962 raft_consensus.cc:348] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:28.841404 19962 raft_consensus.cc:374] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:28.841563 19962 raft_consensus.cc:729] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:28.841918 19962 consensus_queue.cc:260] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:28.843576 19962 ts_tablet_manager.cc:1398] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:14:28.851300 19961 tablet_bootstrap.cc:492] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:28.854272 19962 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:28.857002 19962 tablet_bootstrap.cc:654] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:28.858394 19965 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:28.860580 19961 tablet_bootstrap.cc:654] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:28.860671 19965 tablet_bootstrap.cc:654] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:28.861061 19960 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:28.862987 19965 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:28.863181 19965 ts_tablet_manager.cc:1367] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:14:28.863307 19960 tablet_bootstrap.cc:654] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:28.863343 19961 tablet_bootstrap.cc:492] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:28.863508 19961 ts_tablet_manager.cc:1367] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 14:14:28.864086 19965 raft_consensus.cc:348] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:28.864339 19965 raft_consensus.cc:374] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:28.864265 19961 raft_consensus.cc:348] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.864446 19965 raft_consensus.cc:729] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:28.864482 19961 raft_consensus.cc:374] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:28.864547 19961 raft_consensus.cc:729] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:28.864759 19965 consensus_queue.cc:260] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:28.864826 19961 consensus_queue.cc:260] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.865550 19965 ts_tablet_manager.cc:1398] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:28.865865 19965 tablet_bootstrap.cc:492] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:28.866884 19961 ts_tablet_manager.cc:1398] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:28.868089 19965 tablet_bootstrap.cc:654] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:28.868438 19960 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:28.868597 19960 ts_tablet_manager.cc:1367] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 14:14:28.870112 19962 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:28.870373 19962 ts_tablet_manager.cc:1367] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20240503 14:14:28.870694 19960 raft_consensus.cc:348] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:28.870981 19960 raft_consensus.cc:374] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:28.871090 19960 raft_consensus.cc:729] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:28.871321 19962 raft_consensus.cc:348] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.871606 19962 raft_consensus.cc:374] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:28.871413 19960 consensus_queue.cc:260] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:28.871701 19962 raft_consensus.cc:729] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:28.872203 19960 ts_tablet_manager.cc:1398] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:14:28.872635 19962 consensus_queue.cc:260] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.873302 19960 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:28.873481 19962 ts_tablet_manager.cc:1398] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:14:28.876184 19965 tablet_bootstrap.cc:492] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:28.876400 19965 ts_tablet_manager.cc:1367] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 14:14:28.876716 19960 tablet_bootstrap.cc:654] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:28.877359 19965 raft_consensus.cc:348] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.877620 19965 raft_consensus.cc:374] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:28.877701 19965 raft_consensus.cc:729] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:28.878015 19965 consensus_queue.cc:260] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.878787 19965 ts_tablet_manager.cc:1398] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:28.880201 19962 tablet_bootstrap.cc:492] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:28.882256 19962 tablet_bootstrap.cc:654] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:28.887321 19960 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:28.887519 19960 ts_tablet_manager.cc:1367] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20240503 14:14:28.887697 19962 tablet_bootstrap.cc:492] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:28.887884 19962 ts_tablet_manager.cc:1367] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240503 14:14:28.888324 19960 raft_consensus.cc:348] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.888532 19960 raft_consensus.cc:374] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:28.888600 19960 raft_consensus.cc:729] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:28.888751 19962 raft_consensus.cc:348] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.888933 19962 raft_consensus.cc:374] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:28.888856 19960 consensus_queue.cc:260] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.889031 19962 raft_consensus.cc:729] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:28.889448 19962 consensus_queue.cc:260] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:28.889601 19960 ts_tablet_manager.cc:1398] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:28.890033 19962 ts_tablet_manager.cc:1398] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:28.915663 19848 raft_consensus.cc:482] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:28.915887 19848 raft_consensus.cc:504] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:28.916678 19848 leader_election.cc:288] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:28.917225 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c5c6f4317649fab75d01e339b76403" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:28.917233 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c5c6f4317649fab75d01e339b76403" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:28.917503 18529 raft_consensus.cc:2454] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:28.917513 18399 raft_consensus.cc:2454] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:28.918062 18204 leader_election.cc:302] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:28.918362 19848 raft_consensus.cc:2781] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:28.918494 19848 raft_consensus.cc:482] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:28.918586 19848 raft_consensus.cc:3037] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:28.920610 19848 raft_consensus.cc:504] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:28.921347 19848 leader_election.cc:288] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:28.921895 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c5c6f4317649fab75d01e339b76403" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:28.921974 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c5c6f4317649fab75d01e339b76403" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:28.922168 18529 raft_consensus.cc:3037] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:28.922159 18399 raft_consensus.cc:3037] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:28.925154 18529 raft_consensus.cc:2454] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:28.925132 18399 raft_consensus.cc:2454] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:28.926050 18204 leader_election.cc:302] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:28.926419 19848 raft_consensus.cc:2781] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:28.926579 19848 raft_consensus.cc:686] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:28.926899 19848 consensus_queue.cc:237] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:28.931032 18126 catalog_manager.cc:5484] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:29.090752 19865 raft_consensus.cc:482] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:29.090951 19865 raft_consensus.cc:504] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:29.091758 19865 leader_election.cc:288] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:29.092464 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2167226099a4dcf98304918e8f412af" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838" is_pre_election: true
I20240503 14:14:29.092456 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2167226099a4dcf98304918e8f412af" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:29.092741 18268 raft_consensus.cc:2454] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:29.092742 18399 raft_consensus.cc:2454] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:29.093261 18595 leader_election.cc:302] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:29.093582 19865 raft_consensus.cc:2781] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:29.093704 19865 raft_consensus.cc:482] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:29.093786 19865 raft_consensus.cc:3037] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:29.095803 19865 raft_consensus.cc:504] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:29.096472 19865 leader_election.cc:288] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:29.097198 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2167226099a4dcf98304918e8f412af" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:29.097191 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2167226099a4dcf98304918e8f412af" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:29.097468 18268 raft_consensus.cc:3037] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:29.097468 18399 raft_consensus.cc:3037] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:29.100343 18399 raft_consensus.cc:2454] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:29.100349 18268 raft_consensus.cc:2454] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:29.100932 18595 leader_election.cc:302] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:29.101357 19865 raft_consensus.cc:2781] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:29.101774 19865 raft_consensus.cc:686] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Becoming Leader. State: Replica: 1c72210d694344d4b4604140088a6471, State: Running, Role: LEADER
I20240503 14:14:29.102182 19865 consensus_queue.cc:237] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:29.105572 18126 catalog_manager.cc:5484] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c72210d694344d4b4604140088a6471 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:29.168367 19848 raft_consensus.cc:482] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:29.168605 19848 raft_consensus.cc:504] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:29.169260 19848 leader_election.cc:288] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:29.169927 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29bdb71f24b46b32e7b97d0172c20" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:29.169898 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29bdb71f24b46b32e7b97d0172c20" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:29.170154 18659 raft_consensus.cc:2454] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:29.170181 18529 raft_consensus.cc:2454] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:29.170607 18202 leader_election.cc:302] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:29.170957 19848 raft_consensus.cc:2781] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:29.171068 19848 raft_consensus.cc:482] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:29.171236 19848 raft_consensus.cc:3037] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:29.174299 19848 raft_consensus.cc:504] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:29.175132 19848 leader_election.cc:288] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:29.175663 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29bdb71f24b46b32e7b97d0172c20" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:29.175895 18659 raft_consensus.cc:3037] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:29.176491 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29bdb71f24b46b32e7b97d0172c20" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:29.176723 18529 raft_consensus.cc:3037] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:29.178872 18659 raft_consensus.cc:2454] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:29.179452 18529 raft_consensus.cc:2454] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:29.179524 18202 leader_election.cc:302] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:29.179880 19848 raft_consensus.cc:2781] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:29.180063 19848 raft_consensus.cc:686] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:29.180377 19848 consensus_queue.cc:237] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } }
I20240503 14:14:29.183574 18126 catalog_manager.cc:5484] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:29.379767 19848 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:29.389405 19892 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:29.577358 19865 consensus_queue.cc:1035] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:29.586611 19848 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:29.587903 19865 consensus_queue.cc:1035] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:29.597695 19848 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:29.615842 19977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:29.616206 19977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:29.624058 19977 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 208.043 ms
time per row: 0.104021 ms
I20240503 14:14:29.988811 18125 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49596:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:14:29.999787 18510 tablet_service.cc:1467] Processing CreateTablet for tablet 963ecc30d66b43a4ad149f596bd288d9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:30.000285 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 963ecc30d66b43a4ad149f596bd288d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:30.000854 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 963ecc30d66b43a4ad149f596bd288d9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:30.001363 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 963ecc30d66b43a4ad149f596bd288d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:30.001587 18379 tablet_service.cc:1467] Processing CreateTablet for tablet fe3d20ad69474bcbae19bdd06cbfeb52 (DEFAULT_TABLE table=rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:30.001681 18639 tablet_service.cc:1467] Processing CreateTablet for tablet 963ecc30d66b43a4ad149f596bd288d9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:30.001953 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3d20ad69474bcbae19bdd06cbfeb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:30.002084 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 963ecc30d66b43a4ad149f596bd288d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:30.005793 18638 tablet_service.cc:1467] Processing CreateTablet for tablet 4d24e31ff3ad41d98214bb41d8c28fda (DEFAULT_TABLE table=rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:30.006265 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d24e31ff3ad41d98214bb41d8c28fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:30.009765 18380 tablet_service.cc:1467] Processing CreateTablet for tablet 4d24e31ff3ad41d98214bb41d8c28fda (DEFAULT_TABLE table=rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:30.010252 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d24e31ff3ad41d98214bb41d8c28fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:30.013834 18248 tablet_service.cc:1467] Processing CreateTablet for tablet fe3d20ad69474bcbae19bdd06cbfeb52 (DEFAULT_TABLE table=rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:30.014345 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3d20ad69474bcbae19bdd06cbfeb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:30.019059 18509 tablet_service.cc:1467] Processing CreateTablet for tablet 4d24e31ff3ad41d98214bb41d8c28fda (DEFAULT_TABLE table=rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:30.019577 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d24e31ff3ad41d98214bb41d8c28fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:30.023206 18508 tablet_service.cc:1467] Processing CreateTablet for tablet fe3d20ad69474bcbae19bdd06cbfeb52 (DEFAULT_TABLE table=rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:14:30.023650 18508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3d20ad69474bcbae19bdd06cbfeb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:30.039371 20043 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:30.040412 20042 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:30.042445 20042 tablet_bootstrap.cc:654] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:30.042755 20043 tablet_bootstrap.cc:654] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:30.044116 20041 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:30.045960 20041 tablet_bootstrap.cc:654] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:30.051173 20041 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:30.051270 20044 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838: Bootstrap starting.
I20240503 14:14:30.051371 20041 ts_tablet_manager.cc:1367] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240503 14:14:30.052196 20041 raft_consensus.cc:348] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.052438 20041 raft_consensus.cc:374] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:30.052527 20041 raft_consensus.cc:729] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:30.052783 20041 consensus_queue.cc:260] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.053622 20041 ts_tablet_manager.cc:1398] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:30.053993 20041 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:30.054277 20042 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:30.054430 20042 ts_tablet_manager.cc:1367] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20240503 14:14:30.054731 20043 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:30.054881 20043 ts_tablet_manager.cc:1367] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20240503 14:14:30.055280 20042 raft_consensus.cc:348] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.055472 20042 raft_consensus.cc:374] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:30.055554 20042 raft_consensus.cc:729] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:30.055635 20041 tablet_bootstrap.cc:654] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:30.055665 20043 raft_consensus.cc:348] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.055866 20043 raft_consensus.cc:374] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:30.055819 20042 consensus_queue.cc:260] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.055946 20043 raft_consensus.cc:729] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:30.056165 20043 consensus_queue.cc:260] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.056874 20044 tablet_bootstrap.cc:654] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:30.057822 20042 ts_tablet_manager.cc:1398] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:14:30.058120 20042 tablet_bootstrap.cc:492] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:30.058364 20043 ts_tablet_manager.cc:1398] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 14:14:30.058665 20043 tablet_bootstrap.cc:492] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471: Bootstrap starting.
I20240503 14:14:30.060232 20043 tablet_bootstrap.cc:654] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:30.060968 20042 tablet_bootstrap.cc:654] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:30.069662 20042 tablet_bootstrap.cc:492] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:30.069897 20042 ts_tablet_manager.cc:1367] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 14:14:30.070710 20042 raft_consensus.cc:348] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:30.070878 20044 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838: No bootstrap required, opened a new log
I20240503 14:14:30.070951 20042 raft_consensus.cc:374] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:30.071025 20044 ts_tablet_manager.cc:1367] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.006s
I20240503 14:14:30.071031 20042 raft_consensus.cc:729] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:30.071251 20041 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:30.071419 20043 tablet_bootstrap.cc:492] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471: No bootstrap required, opened a new log
I20240503 14:14:30.071318 20042 consensus_queue.cc:260] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:30.071470 20041 ts_tablet_manager.cc:1367] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.009s
I20240503 14:14:30.071674 20043 ts_tablet_manager.cc:1367] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240503 14:14:30.071975 20044 raft_consensus.cc:348] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.072260 20044 raft_consensus.cc:374] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:30.072252 20041 raft_consensus.cc:348] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.072396 20044 raft_consensus.cc:729] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Initialized, Role: FOLLOWER
I20240503 14:14:30.072459 20043 raft_consensus.cc:348] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:30.072759 20043 raft_consensus.cc:374] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:30.072878 20043 raft_consensus.cc:729] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Initialized, Role: FOLLOWER
I20240503 14:14:30.072801 20044 consensus_queue.cc:260] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.072955 20042 ts_tablet_manager.cc:1398] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:30.073165 20043 consensus_queue.cc:260] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:30.073318 20042 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722: Bootstrap starting.
I20240503 14:14:30.073899 20043 ts_tablet_manager.cc:1398] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:30.072530 20041 raft_consensus.cc:374] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:30.074522 20041 raft_consensus.cc:729] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:30.074826 20041 consensus_queue.cc:260] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.075482 20041 ts_tablet_manager.cc:1398] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:14:30.075851 20041 tablet_bootstrap.cc:492] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568: Bootstrap starting.
I20240503 14:14:30.076611 20042 tablet_bootstrap.cc:654] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:30.077620 20041 tablet_bootstrap.cc:654] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:30.077869 20044 ts_tablet_manager.cc:1398] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:14:30.083382 20041 tablet_bootstrap.cc:492] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568: No bootstrap required, opened a new log
I20240503 14:14:30.083382 20042 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722: No bootstrap required, opened a new log
I20240503 14:14:30.083652 20041 ts_tablet_manager.cc:1367] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240503 14:14:30.084242 20042 ts_tablet_manager.cc:1367] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240503 14:14:30.084969 20041 raft_consensus.cc:348] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:30.085251 20041 raft_consensus.cc:374] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:30.085179 20042 raft_consensus.cc:348] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.085376 20041 raft_consensus.cc:729] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Initialized, Role: FOLLOWER
I20240503 14:14:30.085400 20042 raft_consensus.cc:374] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:30.085480 20042 raft_consensus.cc:729] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Initialized, Role: FOLLOWER
I20240503 14:14:30.085608 20041 consensus_queue.cc:260] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:30.085701 20042 consensus_queue.cc:260] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.086165 20041 ts_tablet_manager.cc:1398] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:30.086854 20042 ts_tablet_manager.cc:1398] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:30.098177 19932 raft_consensus.cc:482] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:30.098363 19932 raft_consensus.cc:504] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.099038 19932 leader_election.cc:288] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:30.099479 18658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "963ecc30d66b43a4ad149f596bd288d9" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471" is_pre_election: true
I20240503 14:14:30.099525 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "963ecc30d66b43a4ad149f596bd288d9" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:30.099694 18658 raft_consensus.cc:2454] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:30.099782 18530 raft_consensus.cc:2454] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 0.
I20240503 14:14:30.100145 18333 leader_election.cc:302] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:30.100432 19932 raft_consensus.cc:2781] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:30.100543 19932 raft_consensus.cc:482] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:30.100636 19932 raft_consensus.cc:3037] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:30.102779 19932 raft_consensus.cc:504] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.103401 19932 leader_election.cc:288] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:30.103873 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "963ecc30d66b43a4ad149f596bd288d9" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:30.103883 18658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "963ecc30d66b43a4ad149f596bd288d9" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:30.104063 18658 raft_consensus.cc:3037] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:30.104072 18530 raft_consensus.cc:3037] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:30.105964 18658 raft_consensus.cc:2454] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:30.105985 18530 raft_consensus.cc:2454] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 1.
I20240503 14:14:30.106388 18333 leader_election.cc:302] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:30.106696 19932 raft_consensus.cc:2781] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:30.107141 19932 raft_consensus.cc:686] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Becoming Leader. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Running, Role: LEADER
I20240503 14:14:30.107471 19932 consensus_queue.cc:237] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.111099 18122 catalog_manager.cc:5484] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:30.248536 19974 raft_consensus.cc:482] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:30.248701 19974 raft_consensus.cc:504] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:30.249437 19974 leader_election.cc:288] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:30.249658 19848 raft_consensus.cc:482] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:14:30.249842 19848 raft_consensus.cc:504] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.249936 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d24e31ff3ad41d98214bb41d8c28fda" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:30.249979 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d24e31ff3ad41d98214bb41d8c28fda" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:30.250152 18530 raft_consensus.cc:2454] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:30.250211 18398 raft_consensus.cc:2454] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 0.
I20240503 14:14:30.250603 18596 leader_election.cc:302] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722; no voters:
I20240503 14:14:30.250722 19848 leader_election.cc:288] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:30.250941 19974 raft_consensus.cc:2781] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:30.251091 19974 raft_consensus.cc:482] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:30.251031 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d20ad69474bcbae19bdd06cbfeb52" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568" is_pre_election: true
I20240503 14:14:30.251219 19974 raft_consensus.cc:3037] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:30.251241 18398 raft_consensus.cc:2454] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:30.251293 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d20ad69474bcbae19bdd06cbfeb52" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722" is_pre_election: true
I20240503 14:14:30.251564 18530 raft_consensus.cc:2454] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 0.
I20240503 14:14:30.251600 18204 leader_election.cc:302] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:30.251856 19848 raft_consensus.cc:2781] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:14:30.251946 19848 raft_consensus.cc:482] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:14:30.252063 19848 raft_consensus.cc:3037] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:30.253482 19974 raft_consensus.cc:504] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:30.254170 19848 raft_consensus.cc:504] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.254454 19974 leader_election.cc:288] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Requested vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:30.254621 18530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d24e31ff3ad41d98214bb41d8c28fda" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:30.254812 19848 leader_election.cc:288] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:30.255020 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d24e31ff3ad41d98214bb41d8c28fda" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:30.255225 18530 raft_consensus.cc:3037] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:30.255255 18398 raft_consensus.cc:3037] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:30.255434 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d20ad69474bcbae19bdd06cbfeb52" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:30.255622 18529 raft_consensus.cc:3037] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:30.257725 18529 raft_consensus.cc:2454] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:30.255261 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3d20ad69474bcbae19bdd06cbfeb52" candidate_uuid: "7ceaaad4bc544c4487859a6671e66838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:30.258049 18530 raft_consensus.cc:2454] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:30.258056 18398 raft_consensus.cc:2454] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 1.
I20240503 14:14:30.258162 18399 raft_consensus.cc:3037] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:30.258535 18595 leader_election.cc:302] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:30.258809 18205 leader_election.cc:302] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:30.258925 19974 raft_consensus.cc:2781] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:30.259094 19974 raft_consensus.cc:686] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Becoming Leader. State: Replica: 1c72210d694344d4b4604140088a6471, State: Running, Role: LEADER
I20240503 14:14:30.259135 19848 raft_consensus.cc:2781] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:30.259382 19974 consensus_queue.cc:237] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } }
I20240503 14:14:30.259685 19848 raft_consensus.cc:686] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:30.259977 19848 consensus_queue.cc:237] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } }
I20240503 14:14:30.260638 18399 raft_consensus.cc:2454] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ceaaad4bc544c4487859a6671e66838 in term 1.
I20240503 14:14:30.263033 18122 catalog_manager.cc:5484] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c72210d694344d4b4604140088a6471 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } }
I20240503 14:14:30.264875 18119 catalog_manager.cc:5484] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:30.615402 19932 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:30.623597 19932 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:14:30.653872 20058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:30.654201 20058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:30.661756 20058 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:14:30.668709 19848 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:30.679538 19848 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240503 14:14:30.783598 19974 consensus_queue.cc:1035] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:30.807349 18399 raft_consensus.cc:1261] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:30.808079 19974 consensus_queue.cc:1035] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:14:30.922276 20062 mvcc.cc:205] Tried to move back new op lower bound from 7023598267619700736 to 7023598265385648128. Current Snapshot: MvccSnapshot[applied={T|T < 7023598267619700736 or (T in {7023598267619700736})}]
INSERT report
rows total: 2000
time total: 322.529 ms
time per row: 0.161264 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:14:32.011296 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.5:40605
--local_ip_for_outbound_sockets=127.10.145.5
--tserver_master_addrs=127.10.145.62:33209
--webserver_port=33243
--webserver_interface=127.10.145.5
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 14:14:32.332419 20126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:32.332672 20126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:32.332823 20126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:32.340039 20126 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:32.340140 20126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:32.340286 20126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.5
I20240503 14:14:32.349776 20126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.5:40605
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.10.145.5
--webserver_port=33243
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.5
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:32.350950 20126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:32.352571 20126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:32.364101 20134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:32.364475 20133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:32.364769 20136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:32.365157 20126 server_base.cc:1034] running on GCE node
I20240503 14:14:32.365722 20126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:32.366555 20126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:32.367755 20126 hybrid_clock.cc:648] HybridClock initialized: now 1714745672367713 us; error 36 us; skew 500 ppm
I20240503 14:14:32.368037 20126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:32.374365 20126 webserver.cc:460] Webserver started at http://127.10.145.5:33243/ using document root <none> and password file <none>
I20240503 14:14:32.375037 20126 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:32.375155 20126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:32.380108 20126 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 14:14:32.382587 20142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:32.383805 20126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:32.383929 20126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2e6a3f22c7ae444e96f6d35cca602e62"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-v549"
I20240503 14:14:32.384685 20126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:32.433495 20126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:32.434279 20126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:32.434517 20126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:32.435364 20126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:32.436748 20126 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:32.436817 20126 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:32.436904 20126 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:32.436944 20126 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:32.465457 20126 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.5:40605
I20240503 14:14:32.465500 20254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.5:40605 every 8 connection(s)
I20240503 14:14:32.466769 20126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:14:32.467334 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 20126
I20240503 14:14:32.468053 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.6:36735
--local_ip_for_outbound_sockets=127.10.145.6
--tserver_master_addrs=127.10.145.62:33209
--webserver_port=38669
--webserver_interface=127.10.145.6
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:32.480149 20255 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:32.480432 20255 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:32.481549 20255 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:32.483155 18122 ts_manager.cc:194] Re-registered known tserver with Master: 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605)
I20240503 14:14:32.483930 18122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.5:58163
W20240503 14:14:32.795653 20259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:32.795898 20259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:32.796028 20259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:32.803663 20259 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:32.803987 20259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:32.804450 20259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.6
I20240503 14:14:32.813256 20259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.6:36735
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.10.145.6
--webserver_port=38669
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.6
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:32.814548 20259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:32.816191 20259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:32.826862 20264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:32.827178 20265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:32.827499 20267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:32.828519 20259 server_base.cc:1034] running on GCE node
I20240503 14:14:32.829108 20259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:32.829996 20259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:32.831187 20259 hybrid_clock.cc:648] HybridClock initialized: now 1714745672831136 us; error 47 us; skew 500 ppm
I20240503 14:14:32.831480 20259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:32.833540 20259 webserver.cc:460] Webserver started at http://127.10.145.6:38669/ using document root <none> and password file <none>
I20240503 14:14:32.834188 20259 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:32.834302 20259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:32.839027 20259 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:14:32.841818 20273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:32.843027 20259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:32.843169 20259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "69cbbbedfe03437b8c95b7bb730dbe66"
format_stamp: "Formatted at 2024-05-03 14:14:09 on dist-test-slave-v549"
I20240503 14:14:32.843890 20259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:32.872349 20259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:32.873092 20259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:32.873384 20259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:32.874243 20259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:32.875674 20259 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:32.875743 20259 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:32.875852 20259 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:32.875897 20259 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:32.903630 20259 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.6:36735
I20240503 14:14:32.903677 20386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.6:36735 every 8 connection(s)
I20240503 14:14:32.904913 20259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:14:32.906344 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 20259
I20240503 14:14:32.906936 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.7:38307
--local_ip_for_outbound_sockets=127.10.145.7
--tserver_master_addrs=127.10.145.62:33209
--webserver_port=33349
--webserver_interface=127.10.145.7
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:32.922137 20387 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:32.922400 20387 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:32.923143 20387 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:32.924472 18122 ts_manager.cc:194] Re-registered known tserver with Master: 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735)
I20240503 14:14:32.925477 18122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.6:60687
W20240503 14:14:33.229866 20391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:33.230103 20391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:33.230228 20391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:33.238503 20391 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:33.238606 20391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:33.238755 20391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.7
I20240503 14:14:33.247236 20391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.7:38307
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.10.145.7
--webserver_port=33349
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.7
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:33.248478 20391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:33.250101 20391 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:33.261520 20398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:33.261749 20400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:33.261936 20397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:33.262591 20391 server_base.cc:1034] running on GCE node
I20240503 14:14:33.263118 20391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:33.263948 20391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:33.265125 20391 hybrid_clock.cc:648] HybridClock initialized: now 1714745673265078 us; error 42 us; skew 500 ppm
I20240503 14:14:33.265501 20391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:33.267818 20391 webserver.cc:460] Webserver started at http://127.10.145.7:33349/ using document root <none> and password file <none>
I20240503 14:14:33.268478 20391 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:33.268584 20391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:33.273880 20391 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:33.276932 20406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:33.278187 20391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:33.278326 20391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a632094ba4554a4895d6c0b538b81af8"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-v549"
I20240503 14:14:33.279071 20391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:33.300359 20391 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:33.300999 20391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:33.301250 20391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:33.302193 20391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:33.303512 20391 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:33.303584 20391 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:33.303701 20391 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:33.303750 20391 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:33.340663 20391 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.7:38307
I20240503 14:14:33.340695 20519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.7:38307 every 8 connection(s)
I20240503 14:14:33.342023 20391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:14:33.348726 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 20391
I20240503 14:14:33.349414 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.8:44107
--local_ip_for_outbound_sockets=127.10.145.8
--tserver_master_addrs=127.10.145.62:33209
--webserver_port=38929
--webserver_interface=127.10.145.8
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:33.353431 20520 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:33.353677 20520 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:33.354422 20520 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:33.355650 18122 ts_manager.cc:194] Re-registered known tserver with Master: a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307)
I20240503 14:14:33.356377 18122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.7:35109
I20240503 14:14:33.486704 20255 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
W20240503 14:14:33.678814 20523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:33.679078 20523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:33.679219 20523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:33.687139 20523 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:33.687271 20523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:33.687446 20523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.8
I20240503 14:14:33.696098 20523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.8:44107
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.10.145.8
--webserver_port=38929
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.8
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:33.697217 20523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:33.699044 20523 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:33.710072 20528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:33.710494 20531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:33.710817 20529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:33.711444 20523 server_base.cc:1034] running on GCE node
I20240503 14:14:33.711961 20523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:33.713161 20523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:33.714375 20523 hybrid_clock.cc:648] HybridClock initialized: now 1714745673714325 us; error 61 us; skew 500 ppm
I20240503 14:14:33.714666 20523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:33.716629 20523 webserver.cc:460] Webserver started at http://127.10.145.8:38929/ using document root <none> and password file <none>
I20240503 14:14:33.717236 20523 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:33.717375 20523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:33.722612 20523 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 14:14:33.725359 20537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:33.726598 20523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:33.726737 20523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "80260781ae364d049c2500ff76a06ce8"
format_stamp: "Formatted at 2024-05-03 14:14:10 on dist-test-slave-v549"
I20240503 14:14:33.727560 20523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:33.750770 20523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:33.751399 20523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:33.751614 20523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:33.752355 20523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:33.753695 20523 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:33.753755 20523 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:33.753859 20523 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:33.753903 20523 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:33.787757 20523 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.8:44107
I20240503 14:14:33.787796 20649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.8:44107 every 8 connection(s)
I20240503 14:14:33.789041 20523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:14:33.795917 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 20523
I20240503 14:14:33.801892 20650 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:33.802143 20650 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:33.802841 20650 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:33.804139 18122 ts_manager.cc:194] Re-registered known tserver with Master: 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107)
I20240503 14:14:33.805083 18122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.8:52077
I20240503 14:14:33.927752 20387 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
W20240503 14:14:34.148983 20655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:34.149274 20655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:34.156615 20655 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:14:34.358696 20520 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
I20240503 14:14:34.408015 18639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:34.415382 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:34.419876 18510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:34.450887 18639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:34.462599 20585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:34.489956 20322 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:34.491077 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:34.495503 18248 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:14:34.543051 18510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:34.557212 20322 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:34.561232 18378 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:34.582201 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:34.611572 18248 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:14:34.625910 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:34.641825 20585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:34.674861 18378 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:34.807543 20650 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
I20240503 14:14:34.933260 18639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:34.953644 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.004843 18510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:35.025841 20322 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.040452 18248 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:14:35.091799 20585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.101099 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.124491 18378 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:14:35.400498 20785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:35.400985 20785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:35.412739 20785 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:14:35.414026 18639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:35.451860 18510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:35.456254 18248 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:14:35.457485 20585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.461721 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.484922 18378 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:35.502168 20322 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.536586 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.673785 18399 consensus_queue.cc:237] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:35.678777 18267 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:35.679728 20392 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:35.684603 18528 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:35.685307 20870 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:35.687218 20392 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:35.696985 18267 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
W20240503 14:14:35.701040 18334 consensus_peers.cc:485] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 -> Peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Couldn't send request to peer 69cbbbedfe03437b8c95b7bb730dbe66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0c1a7f5739f4fa59271254e501fb095. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:35.723774 18120 catalog_manager.cc:5484] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:35.741730 18528 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:35.764096 18399 consensus_queue.cc:237] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:35.777323 18639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:35.803843 18267 raft_consensus.cc:1261] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:35.804679 20870 consensus_queue.cc:1035] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:35.812687 18528 raft_consensus.cc:1261] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:35.813702 20875 consensus_queue.cc:1035] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:35.819118 18336 consensus_peers.cc:485] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 -> Peer a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307): Couldn't send request to peer a632094ba4554a4895d6c0b538b81af8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d907591e3c9740e8b5eee42f8d41996b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:35.837112 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.859745 20322 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.876767 20392 raft_consensus.cc:2932] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:35.877568 18529 raft_consensus.cc:2932] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:35.885272 18125 catalog_manager.cc:5484] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:35.912801 18267 raft_consensus.cc:2932] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:35.915046 18510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:35.944810 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.959141 20585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:35.960175 18248 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:14:35.967609 18378 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:35.984372 18267 consensus_queue.cc:237] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:36.014328 18658 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.015228 20845 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.019335 20845 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:36.020397 18399 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.021185 20717 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.026738 18658 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:36.027338 18125 catalog_manager.cc:5484] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:36.038100 18399 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
W20240503 14:14:36.056514 18202 consensus_peers.cc:485] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 -> Peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107): Couldn't send request to peer 80260781ae364d049c2500ff76a06ce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91743575cf4a4f38b5b43265539237a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.087846 18399 consensus_queue.cc:237] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:36.090939 18658 raft_consensus.cc:1261] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.090957 18267 raft_consensus.cc:1261] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.091490 20392 consensus_queue.cc:1035] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.091820 20875 consensus_queue.cc:1035] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.097124 20392 raft_consensus.cc:2932] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.098630 18658 raft_consensus.cc:2932] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
W20240503 14:14:36.099834 18336 consensus_peers.cc:485] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 -> Peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Couldn't send request to peer 2e6a3f22c7ae444e96f6d35cca602e62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a23a38ec48148b19bf8102a944ab11e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.103870 18125 catalog_manager.cc:5484] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:36.116271 18267 raft_consensus.cc:2932] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
Runtime error: ksck discovered errors
I20240503 14:14:36.142884 18267 consensus_queue.cc:237] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:36.158253 18399 raft_consensus.cc:1261] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.158262 18528 raft_consensus.cc:1261] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.159142 20845 consensus_queue.cc:1035] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.159737 20845 consensus_queue.cc:1035] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:36.165656 18203 consensus_peers.cc:485] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 -> Peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Couldn't send request to peer 69cbbbedfe03437b8c95b7bb730dbe66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c18cad62ab6443cea66d08f0980c75b6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.167129 20845 raft_consensus.cc:2932] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:36.168495 18528 raft_consensus.cc:2932] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:36.170717 18399 raft_consensus.cc:2932] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:36.173260 18119 catalog_manager.cc:5484] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:36.198002 20932 ts_tablet_manager.cc:902] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: Initiating tablet copy from peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:36.197918 18399 consensus_queue.cc:237] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:36.201023 18267 raft_consensus.cc:1261] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:36.201857 18336 consensus_peers.cc:485] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 -> Peer a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307): Couldn't send request to peer a632094ba4554a4895d6c0b538b81af8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87fded61bfc240f5846080c328528edc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.201215 18528 raft_consensus.cc:1261] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.202540 20875 consensus_queue.cc:1035] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.202934 20870 consensus_queue.cc:1035] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.215368 20870 raft_consensus.cc:2932] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.218533 18267 raft_consensus.cc:2932] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.218523 18528 raft_consensus.cc:2932] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.223932 18125 catalog_manager.cc:5484] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:36.227777 20932 tablet_copy_client.cc:308] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:43191
I20240503 14:14:36.240797 18420 tablet_copy_service.cc:140] P ea366e6e15a24c52b1ea5a154f942568: Received BeginTabletCopySession request for tablet d907591e3c9740e8b5eee42f8d41996b from peer a632094ba4554a4895d6c0b538b81af8 ({username='slave'} at 127.10.145.7:49165)
I20240503 14:14:36.241083 18420 tablet_copy_service.cc:161] P ea366e6e15a24c52b1ea5a154f942568: Beginning new tablet copy session on tablet d907591e3c9740e8b5eee42f8d41996b from peer a632094ba4554a4895d6c0b538b81af8 at {username='slave'} at 127.10.145.7:49165: session id = a632094ba4554a4895d6c0b538b81af8-d907591e3c9740e8b5eee42f8d41996b
I20240503 14:14:36.244015 18420 tablet_copy_source_session.cc:205] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:36.250290 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d907591e3c9740e8b5eee42f8d41996b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.254009 18267 consensus_queue.cc:237] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:36.256930 18658 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.257155 18528 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.257627 20845 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:36.258076 18202 consensus_peers.cc:485] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 -> Peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107): Couldn't send request to peer 80260781ae364d049c2500ff76a06ce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea3ea2e3d87f4c099abbf5193c239a8b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.258366 20905 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.260499 20845 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:36.263731 18659 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:36.267289 18125 catalog_manager.cc:5484] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:36.269335 20932 tablet_copy_client.cc:783] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:36.269716 20932 tablet_copy_client.cc:647] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:36.271296 20949 ts_tablet_manager.cc:902] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: Initiating tablet copy from peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:36.277637 20939 tablet_copy_client.cc:1002] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:36.288772 18528 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:36.301254 18267 consensus_queue.cc:237] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:36.301556 20949 tablet_copy_client.cc:308] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:43191
I20240503 14:14:36.303920 18659 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.304630 18399 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.304678 20717 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.306912 20717 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.310530 18420 tablet_copy_service.cc:140] P ea366e6e15a24c52b1ea5a154f942568: Received BeginTabletCopySession request for tablet f0c1a7f5739f4fa59271254e501fb095 from peer 69cbbbedfe03437b8c95b7bb730dbe66 ({username='slave'} at 127.10.145.6:49369)
I20240503 14:14:36.310739 18420 tablet_copy_service.cc:161] P ea366e6e15a24c52b1ea5a154f942568: Beginning new tablet copy session on tablet f0c1a7f5739f4fa59271254e501fb095 from peer 69cbbbedfe03437b8c95b7bb730dbe66 at {username='slave'} at 127.10.145.6:49369: session id = 69cbbbedfe03437b8c95b7bb730dbe66-f0c1a7f5739f4fa59271254e501fb095
I20240503 14:14:36.313661 18420 tablet_copy_source_session.cc:205] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 14:14:36.314005 18205 consensus_peers.cc:485] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 -> Peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Couldn't send request to peer 2e6a3f22c7ae444e96f6d35cca602e62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eeb40a038c4d404d9cb8ea693d9071d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.316632 20905 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.320053 20949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c1a7f5739f4fa59271254e501fb095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.320348 18659 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.320360 18399 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.326577 18125 catalog_manager.cc:5484] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.341804 20949 tablet_copy_client.cc:783] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:36.342253 20949 tablet_copy_client.cc:647] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:36.344471 20959 tablet_copy_client.cc:1002] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:36.369122 18528 consensus_queue.cc:237] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:36.373088 18399 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.373515 18659 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.374365 20652 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.375115 20652 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.391039 20791 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.393846 18400 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.400889 18120 catalog_manager.cc:5484] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.411993 18659 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.414749 18400 consensus_queue.cc:237] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
W20240503 14:14:36.422267 18334 consensus_peers.cc:485] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 -> Peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Couldn't send request to peer 69cbbbedfe03437b8c95b7bb730dbe66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 963ecc30d66b43a4ad149f596bd288d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.422232 18659 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.423099 18528 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.423099 20870 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.423990 20870 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.426957 20392 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:36.427604 18659 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:36.428460 20979 ts_tablet_manager.cc:902] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:36.433557 18120 catalog_manager.cc:5484] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
W20240503 14:14:36.433964 18466 consensus_peers.cc:485] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 -> Peer a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307): Couldn't send request to peer a632094ba4554a4895d6c0b538b81af8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c63b35eef494f0e8928c4ca0f74ec32. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.434372 18528 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:36.453897 20979 tablet_copy_client.cc:308] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:36.477756 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet 91743575cf4a4f38b5b43265539237a3 from peer 80260781ae364d049c2500ff76a06ce8 ({username='slave'} at 127.10.145.8:59415)
I20240503 14:14:36.478065 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet 91743575cf4a4f38b5b43265539237a3 from peer 80260781ae364d049c2500ff76a06ce8 at {username='slave'} at 127.10.145.8:59415: session id = 80260781ae364d049c2500ff76a06ce8-91743575cf4a4f38b5b43265539237a3
I20240503 14:14:36.481886 18289 tablet_copy_source_session.cc:205] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:36.481976 18659 consensus_queue.cc:237] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:36.488926 20979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91743575cf4a4f38b5b43265539237a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.489563 18528 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.490545 20918 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.493839 18267 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.494412 20918 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.496661 20788 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:36.499873 18267 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:36.503094 18528 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:36.506031 18120 catalog_manager.cc:5484] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
W20240503 14:14:36.529110 18593 consensus_peers.cc:485] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 -> Peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107): Couldn't send request to peer 80260781ae364d049c2500ff76a06ce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7570d5bce0e5481d8dded653617be5a7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.548589 20979 tablet_copy_client.cc:783] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:36.549151 20979 tablet_copy_client.cc:647] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:36.552031 20987 tablet_copy_client.cc:1002] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:36.578106 18528 consensus_queue.cc:237] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:36.587038 18267 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.587630 18659 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:36.588233 18466 consensus_peers.cc:485] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 -> Peer a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307): Couldn't send request to peer a632094ba4554a4895d6c0b538b81af8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f9705f607a94835ad7e6c20668b80ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.588675 20791 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.589884 20652 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.595788 20970 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.596849 18267 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.598708 18659 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.602840 18120 catalog_manager.cc:5484] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.621114 18659 consensus_queue.cc:237] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:36.624147 18400 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.625809 20788 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.628284 18528 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.631069 20918 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:36.640482 18596 consensus_peers.cc:485] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 -> Peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Couldn't send request to peer 2e6a3f22c7ae444e96f6d35cca602e62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9577e6d3801f42f89afe8a48a77031dd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.668699 21014 ts_tablet_manager.cc:902] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: Initiating tablet copy from peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:36.668748 20788 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.669721 21016 ts_tablet_manager.cc:902] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: Initiating tablet copy from peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:36.670176 18529 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.675311 18119 catalog_manager.cc:5484] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:36.687984 18400 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.699920 21016 tablet_copy_client.cc:308] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:43191
I20240503 14:14:36.701452 18420 tablet_copy_service.cc:140] P ea366e6e15a24c52b1ea5a154f942568: Received BeginTabletCopySession request for tablet 87fded61bfc240f5846080c328528edc from peer a632094ba4554a4895d6c0b538b81af8 ({username='slave'} at 127.10.145.7:49165)
I20240503 14:14:36.701635 18420 tablet_copy_service.cc:161] P ea366e6e15a24c52b1ea5a154f942568: Beginning new tablet copy session on tablet 87fded61bfc240f5846080c328528edc from peer a632094ba4554a4895d6c0b538b81af8 at {username='slave'} at 127.10.145.7:49165: session id = a632094ba4554a4895d6c0b538b81af8-87fded61bfc240f5846080c328528edc
I20240503 14:14:36.701987 21014 tablet_copy_client.cc:308] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:43191
I20240503 14:14:36.704545 18420 tablet_copy_source_session.cc:205] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:36.706638 21016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fded61bfc240f5846080c328528edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.711268 18267 consensus_queue.cc:237] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:36.714637 21016 tablet_copy_client.cc:783] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:36.715013 21016 tablet_copy_client.cc:647] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:36.716795 18400 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.716836 18529 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:36.716995 18203 consensus_peers.cc:485] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 -> Peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Couldn't send request to peer 69cbbbedfe03437b8c95b7bb730dbe66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a39fbf1d8f6f46ccab3c2b992ed0c63e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.717449 20905 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.717803 20845 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.722375 21029 tablet_copy_client.cc:1002] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:36.734254 21032 ts_tablet_manager.cc:902] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:36.738830 18420 tablet_copy_service.cc:140] P ea366e6e15a24c52b1ea5a154f942568: Received BeginTabletCopySession request for tablet 5a23a38ec48148b19bf8102a944ab11e from peer 2e6a3f22c7ae444e96f6d35cca602e62 ({username='slave'} at 127.10.145.5:47647)
I20240503 14:14:36.739094 18420 tablet_copy_service.cc:161] P ea366e6e15a24c52b1ea5a154f942568: Beginning new tablet copy session on tablet 5a23a38ec48148b19bf8102a944ab11e from peer 2e6a3f22c7ae444e96f6d35cca602e62 at {username='slave'} at 127.10.145.5:47647: session id = 2e6a3f22c7ae444e96f6d35cca602e62-5a23a38ec48148b19bf8102a944ab11e
I20240503 14:14:36.743108 18420 tablet_copy_source_session.cc:205] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:36.749104 21014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a23a38ec48148b19bf8102a944ab11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.760653 21039 ts_tablet_manager.cc:902] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:36.761756 20845 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:36.765909 21032 tablet_copy_client.cc:308] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:36.766795 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet ea3ea2e3d87f4c099abbf5193c239a8b from peer 80260781ae364d049c2500ff76a06ce8 ({username='slave'} at 127.10.145.8:59415)
I20240503 14:14:36.766361 18528 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:36.766994 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet ea3ea2e3d87f4c099abbf5193c239a8b from peer 80260781ae364d049c2500ff76a06ce8 at {username='slave'} at 127.10.145.8:59415: session id = 80260781ae364d049c2500ff76a06ce8-ea3ea2e3d87f4c099abbf5193c239a8b
I20240503 14:14:36.770524 18289 tablet_copy_source_session.cc:205] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:36.773317 21032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3ea2e3d87f4c099abbf5193c239a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.773070 18125 catalog_manager.cc:5484] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:36.773543 21033 ts_tablet_manager.cc:902] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:36.775960 21014 tablet_copy_client.cc:783] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:36.776403 21014 tablet_copy_client.cc:647] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:36.781924 21028 tablet_copy_client.cc:1002] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:36.785826 18399 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:36.789244 21032 tablet_copy_client.cc:783] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:36.789623 21032 tablet_copy_client.cc:647] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:36.795117 21040 tablet_copy_client.cc:1002] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:36.801159 21033 tablet_copy_client.cc:308] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:36.809335 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet c18cad62ab6443cea66d08f0980c75b6 from peer 69cbbbedfe03437b8c95b7bb730dbe66 ({username='slave'} at 127.10.145.6:41609)
I20240503 14:14:36.809525 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet c18cad62ab6443cea66d08f0980c75b6 from peer 69cbbbedfe03437b8c95b7bb730dbe66 at {username='slave'} at 127.10.145.6:41609: session id = 69cbbbedfe03437b8c95b7bb730dbe66-c18cad62ab6443cea66d08f0980c75b6
I20240503 14:14:36.812474 18289 tablet_copy_source_session.cc:205] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:36.813546 21039 tablet_copy_client.cc:308] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:36.818776 21033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18cad62ab6443cea66d08f0980c75b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.833045 21033 tablet_copy_client.cc:783] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:36.833508 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet eeb40a038c4d404d9cb8ea693d9071d7 from peer 2e6a3f22c7ae444e96f6d35cca602e62 ({username='slave'} at 127.10.145.5:45257)
I20240503 14:14:36.833711 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet eeb40a038c4d404d9cb8ea693d9071d7 from peer 2e6a3f22c7ae444e96f6d35cca602e62 at {username='slave'} at 127.10.145.5:45257: session id = 2e6a3f22c7ae444e96f6d35cca602e62-eeb40a038c4d404d9cb8ea693d9071d7
I20240503 14:14:36.835261 21033 tablet_copy_client.cc:647] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:36.836251 18289 tablet_copy_source_session.cc:205] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:36.837720 21039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb40a038c4d404d9cb8ea693d9071d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:36.840157 21050 tablet_copy_client.cc:1002] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:36.842110 18267 consensus_queue.cc:237] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:36.844619 18659 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.844178 21039 tablet_copy_client.cc:783] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 0 data blocks...
W20240503 14:14:36.844907 18205 consensus_peers.cc:485] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 -> Peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Couldn't send request to peer 2e6a3f22c7ae444e96f6d35cca602e62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79373bd79a934eaab67bbcf2208e7b5b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.845041 21039 tablet_copy_client.cc:647] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:36.845203 18400 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.845815 21042 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.846204 20845 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.846865 21052 tablet_copy_client.cc:1002] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:36.851075 21042 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.851851 18400 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.857904 18119 catalog_manager.cc:5484] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.864107 18659 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:36.883272 18400 consensus_queue.cc:237] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:36.887898 21066 ts_tablet_manager.cc:902] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: Initiating tablet copy from peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:36.888576 18267 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.888636 18659 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.889158 20870 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.889855 20870 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.893390 20870 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
W20240503 14:14:36.897768 18333 consensus_peers.cc:485] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 -> Peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107): Couldn't send request to peer 80260781ae364d049c2500ff76a06ce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c337ef0405e4856bd2a608a6a602a49. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.897491 18267 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:36.902061 18659 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:36.902113 18119 catalog_manager.cc:5484] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:36.940448 21066 tablet_copy_client.cc:308] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:43191
I20240503 14:14:36.941490 18420 tablet_copy_service.cc:140] P ea366e6e15a24c52b1ea5a154f942568: Received BeginTabletCopySession request for tablet 963ecc30d66b43a4ad149f596bd288d9 from peer 69cbbbedfe03437b8c95b7bb730dbe66 ({username='slave'} at 127.10.145.6:49369)
I20240503 14:14:36.941694 18420 tablet_copy_service.cc:161] P ea366e6e15a24c52b1ea5a154f942568: Beginning new tablet copy session on tablet 963ecc30d66b43a4ad149f596bd288d9 from peer 69cbbbedfe03437b8c95b7bb730dbe66 at {username='slave'} at 127.10.145.6:49369: session id = 69cbbbedfe03437b8c95b7bb730dbe66-963ecc30d66b43a4ad149f596bd288d9
W20240503 14:14:36.944576 20958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:36.944877 20958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:14:36.944885 18420 tablet_copy_source_session.cc:205] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:36.948635 21066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 963ecc30d66b43a4ad149f596bd288d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:14:36.956784 20958 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:14:36.956594 18659 consensus_queue.cc:237] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:36.957600 21066 tablet_copy_client.cc:783] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:36.957960 21066 tablet_copy_client.cc:647] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:36.960733 18400 raft_consensus.cc:1261] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.961938 18529 raft_consensus.cc:1261] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.963042 21012 consensus_queue.cc:1035] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.964962 21080 tablet_copy_client.cc:1002] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:36.966409 21012 consensus_queue.cc:1035] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:36.970440 18596 consensus_peers.cc:485] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 -> Peer a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307): Couldn't send request to peer a632094ba4554a4895d6c0b538b81af8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d24e31ff3ad41d98214bb41d8c28fda. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.975889 21083 ts_tablet_manager.cc:902] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: Initiating tablet copy from peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:36.975867 21012 raft_consensus.cc:2932] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.979729 18529 raft_consensus.cc:2932] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.979784 18400 raft_consensus.cc:2932] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:36.986562 18119 catalog_manager.cc:5484] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:37.020615 21083 tablet_copy_client.cc:308] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:38857
I20240503 14:14:37.043391 21099 ts_tablet_manager.cc:902] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:37.055528 18550 tablet_copy_service.cc:140] P 55339e38e7aa4bb7b7b42aa206108722: Received BeginTabletCopySession request for tablet 7c63b35eef494f0e8928c4ca0f74ec32 from peer a632094ba4554a4895d6c0b538b81af8 ({username='slave'} at 127.10.145.7:40207)
I20240503 14:14:37.055843 18550 tablet_copy_service.cc:161] P 55339e38e7aa4bb7b7b42aa206108722: Beginning new tablet copy session on tablet 7c63b35eef494f0e8928c4ca0f74ec32 from peer a632094ba4554a4895d6c0b538b81af8 at {username='slave'} at 127.10.145.7:40207: session id = a632094ba4554a4895d6c0b538b81af8-7c63b35eef494f0e8928c4ca0f74ec32
I20240503 14:14:37.059799 18550 tablet_copy_source_session.cc:205] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.062585 21083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c63b35eef494f0e8928c4ca0f74ec32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.071200 21083 tablet_copy_client.cc:783] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.071805 21083 tablet_copy_client.cc:647] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.074739 21094 tablet_copy_client.cc:1002] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.077250 21099 tablet_copy_client.cc:308] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:37.080629 21110 ts_tablet_manager.cc:902] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: Initiating tablet copy from peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:37.085839 18680 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet 7570d5bce0e5481d8dded653617be5a7 from peer 80260781ae364d049c2500ff76a06ce8 ({username='slave'} at 127.10.145.8:39351)
I20240503 14:14:37.086179 18680 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet 7570d5bce0e5481d8dded653617be5a7 from peer 80260781ae364d049c2500ff76a06ce8 at {username='slave'} at 127.10.145.8:39351: session id = 80260781ae364d049c2500ff76a06ce8-7570d5bce0e5481d8dded653617be5a7
I20240503 14:14:37.087677 18267 consensus_queue.cc:237] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:37.090409 18680 tablet_copy_source_session.cc:205] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.092761 21099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7570d5bce0e5481d8dded653617be5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.100092 21099 tablet_copy_client.cc:783] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.100502 21099 tablet_copy_client.cc:647] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.103935 21108 tablet_copy_client.cc:1002] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.106343 18529 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:37.106985 18400 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:37.107178 18205 consensus_peers.cc:485] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 -> Peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Couldn't send request to peer 2e6a3f22c7ae444e96f6d35cca602e62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eedc3cfb2dff4f9084446d89b9b3c0f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:37.107389 21042 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.107753 20845 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.111074 20845 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:37.111552 18400 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:37.116830 21110 tablet_copy_client.cc:308] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:38857
I20240503 14:14:37.117647 18550 tablet_copy_service.cc:140] P 55339e38e7aa4bb7b7b42aa206108722: Received BeginTabletCopySession request for tablet 7f9705f607a94835ad7e6c20668b80ad from peer a632094ba4554a4895d6c0b538b81af8 ({username='slave'} at 127.10.145.7:40207)
I20240503 14:14:37.117420 18529 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:37.117832 18550 tablet_copy_service.cc:161] P 55339e38e7aa4bb7b7b42aa206108722: Beginning new tablet copy session on tablet 7f9705f607a94835ad7e6c20668b80ad from peer a632094ba4554a4895d6c0b538b81af8 at {username='slave'} at 127.10.145.7:40207: session id = a632094ba4554a4895d6c0b538b81af8-7f9705f607a94835ad7e6c20668b80ad
I20240503 14:14:37.120592 18119 catalog_manager.cc:5484] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:37.122246 18550 tablet_copy_source_session.cc:205] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.125723 21110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9705f607a94835ad7e6c20668b80ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.133065 21110 tablet_copy_client.cc:783] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.133471 21110 tablet_copy_client.cc:647] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.135288 21117 tablet_copy_client.cc:1002] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.146512 18529 consensus_queue.cc:237] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:37.149567 18267 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:37.149637 18659 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:37.150528 20652 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.151471 20652 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.162297 20652 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:37.166226 18267 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:37.168210 18659 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:37.176015 18125 catalog_manager.cc:5484] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
W20240503 14:14:37.181756 18464 consensus_peers.cc:485] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 -> Peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Couldn't send request to peer 69cbbbedfe03437b8c95b7bb730dbe66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19b1b6be3bda47829f53765282eae9f9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:37.198244 18529 consensus_queue.cc:237] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:37.201365 18659 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:37.201567 18267 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:37.202167 20791 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.202545 20652 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.206637 21128 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:37.209523 18659 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:37.212345 18267 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:37.212467 18120 catalog_manager.cc:5484] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:14:37.214146 18463 consensus_peers.cc:485] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 -> Peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107): Couldn't send request to peer 80260781ae364d049c2500ff76a06ce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c35376539bc944e59b74bfe59d5feb9e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:37.221127 21148 ts_tablet_manager.cc:902] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:37.223145 21148 tablet_copy_client.cc:308] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:37.231789 18680 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet 9577e6d3801f42f89afe8a48a77031dd from peer 2e6a3f22c7ae444e96f6d35cca602e62 ({username='slave'} at 127.10.145.5:43483)
I20240503 14:14:37.232026 18680 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet 9577e6d3801f42f89afe8a48a77031dd from peer 2e6a3f22c7ae444e96f6d35cca602e62 at {username='slave'} at 127.10.145.5:43483: session id = 2e6a3f22c7ae444e96f6d35cca602e62-9577e6d3801f42f89afe8a48a77031dd
I20240503 14:14:37.234797 18680 tablet_copy_source_session.cc:205] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.237984 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9577e6d3801f42f89afe8a48a77031dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.247759 21148 tablet_copy_client.cc:783] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.248178 21148 tablet_copy_client.cc:647] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.250463 21154 tablet_copy_client.cc:1002] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.266275 21167 ts_tablet_manager.cc:902] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: Initiating tablet copy from peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:37.276726 21167 tablet_copy_client.cc:308] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:43191
I20240503 14:14:37.295133 18420 tablet_copy_service.cc:140] P ea366e6e15a24c52b1ea5a154f942568: Received BeginTabletCopySession request for tablet 8c337ef0405e4856bd2a608a6a602a49 from peer 80260781ae364d049c2500ff76a06ce8 ({username='slave'} at 127.10.145.8:41697)
I20240503 14:14:37.295346 18420 tablet_copy_service.cc:161] P ea366e6e15a24c52b1ea5a154f942568: Beginning new tablet copy session on tablet 8c337ef0405e4856bd2a608a6a602a49 from peer 80260781ae364d049c2500ff76a06ce8 at {username='slave'} at 127.10.145.8:41697: session id = 80260781ae364d049c2500ff76a06ce8-8c337ef0405e4856bd2a608a6a602a49
I20240503 14:14:37.298105 18420 tablet_copy_source_session.cc:205] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.299556 21167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c337ef0405e4856bd2a608a6a602a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.315627 21167 tablet_copy_client.cc:783] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.315940 21167 tablet_copy_client.cc:647] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.318126 21172 tablet_copy_client.cc:1002] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.318392 21182 ts_tablet_manager.cc:902] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:37.328598 21182 tablet_copy_client.cc:308] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:37.329329 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e from peer 69cbbbedfe03437b8c95b7bb730dbe66 ({username='slave'} at 127.10.145.6:41609)
I20240503 14:14:37.329499 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet a39fbf1d8f6f46ccab3c2b992ed0c63e from peer 69cbbbedfe03437b8c95b7bb730dbe66 at {username='slave'} at 127.10.145.6:41609: session id = 69cbbbedfe03437b8c95b7bb730dbe66-a39fbf1d8f6f46ccab3c2b992ed0c63e
I20240503 14:14:37.332624 18289 tablet_copy_source_session.cc:205] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.335423 21182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.346138 21182 tablet_copy_client.cc:783] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.346475 21182 tablet_copy_client.cc:647] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.348896 21186 tablet_copy_client.cc:1002] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.367429 18639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:37.407480 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:37.432107 18510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:37.432586 20322 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:37.469241 20585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:37.475287 18248 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:14:37.571664 21215 ts_tablet_manager.cc:902] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: Initiating tablet copy from peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:37.592844 21211 ts_tablet_manager.cc:902] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:37.597661 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:37.628351 21217 ts_tablet_manager.cc:902] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:37.630038 18378 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:37.665563 21211 tablet_copy_client.cc:308] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:37.666745 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet 79373bd79a934eaab67bbcf2208e7b5b from peer 2e6a3f22c7ae444e96f6d35cca602e62 ({username='slave'} at 127.10.145.5:45257)
I20240503 14:14:37.666958 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet 79373bd79a934eaab67bbcf2208e7b5b from peer 2e6a3f22c7ae444e96f6d35cca602e62 at {username='slave'} at 127.10.145.5:45257: session id = 2e6a3f22c7ae444e96f6d35cca602e62-79373bd79a934eaab67bbcf2208e7b5b
I20240503 14:14:37.670707 18289 tablet_copy_source_session.cc:205] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.674685 21211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79373bd79a934eaab67bbcf2208e7b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.684484 21215 tablet_copy_client.cc:308] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:38857
I20240503 14:14:37.691486 21211 tablet_copy_client.cc:783] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.692440 21211 tablet_copy_client.cc:647] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.694532 21225 tablet_copy_client.cc:1002] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.696802 21235 ts_tablet_manager.cc:902] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:37.697809 21217 tablet_copy_client.cc:308] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:37.699731 21227 ts_tablet_manager.cc:902] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: Initiating tablet copy from peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:37.720047 18550 tablet_copy_service.cc:140] P 55339e38e7aa4bb7b7b42aa206108722: Received BeginTabletCopySession request for tablet 19b1b6be3bda47829f53765282eae9f9 from peer 69cbbbedfe03437b8c95b7bb730dbe66 ({username='slave'} at 127.10.145.6:38759)
I20240503 14:14:37.720254 18550 tablet_copy_service.cc:161] P 55339e38e7aa4bb7b7b42aa206108722: Beginning new tablet copy session on tablet 19b1b6be3bda47829f53765282eae9f9 from peer 69cbbbedfe03437b8c95b7bb730dbe66 at {username='slave'} at 127.10.145.6:38759: session id = 69cbbbedfe03437b8c95b7bb730dbe66-19b1b6be3bda47829f53765282eae9f9
I20240503 14:14:37.723151 18550 tablet_copy_source_session.cc:205] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.725703 21215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b1b6be3bda47829f53765282eae9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.734774 21215 tablet_copy_client.cc:783] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.735147 21215 tablet_copy_client.cc:647] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.737015 21232 tablet_copy_client.cc:1002] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.739972 21235 tablet_copy_client.cc:308] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:37.740849 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet eedc3cfb2dff4f9084446d89b9b3c0f4 from peer 2e6a3f22c7ae444e96f6d35cca602e62 ({username='slave'} at 127.10.145.5:45257)
I20240503 14:14:37.741020 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet eedc3cfb2dff4f9084446d89b9b3c0f4 from peer 2e6a3f22c7ae444e96f6d35cca602e62 at {username='slave'} at 127.10.145.5:45257: session id = 2e6a3f22c7ae444e96f6d35cca602e62-eedc3cfb2dff4f9084446d89b9b3c0f4
I20240503 14:14:37.744056 18289 tablet_copy_source_session.cc:205] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.744210 18680 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet 4d24e31ff3ad41d98214bb41d8c28fda from peer a632094ba4554a4895d6c0b538b81af8 ({username='slave'} at 127.10.145.7:47505)
I20240503 14:14:37.744372 18680 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet 4d24e31ff3ad41d98214bb41d8c28fda from peer a632094ba4554a4895d6c0b538b81af8 at {username='slave'} at 127.10.145.7:47505: session id = a632094ba4554a4895d6c0b538b81af8-4d24e31ff3ad41d98214bb41d8c28fda
I20240503 14:14:37.747668 18680 tablet_copy_source_session.cc:205] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.750475 21217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d24e31ff3ad41d98214bb41d8c28fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.757095 21227 tablet_copy_client.cc:308] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:38857
I20240503 14:14:37.758697 21217 tablet_copy_client.cc:783] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.758987 21217 tablet_copy_client.cc:647] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.760198 21235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedc3cfb2dff4f9084446d89b9b3c0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.762106 21234 tablet_copy_client.cc:1002] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.765472 18550 tablet_copy_service.cc:140] P 55339e38e7aa4bb7b7b42aa206108722: Received BeginTabletCopySession request for tablet c35376539bc944e59b74bfe59d5feb9e from peer 80260781ae364d049c2500ff76a06ce8 ({username='slave'} at 127.10.145.8:40531)
I20240503 14:14:37.765697 18550 tablet_copy_service.cc:161] P 55339e38e7aa4bb7b7b42aa206108722: Beginning new tablet copy session on tablet c35376539bc944e59b74bfe59d5feb9e from peer 80260781ae364d049c2500ff76a06ce8 at {username='slave'} at 127.10.145.8:40531: session id = 80260781ae364d049c2500ff76a06ce8-c35376539bc944e59b74bfe59d5feb9e
I20240503 14:14:37.766294 21235 tablet_copy_client.cc:783] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.766536 21235 tablet_copy_client.cc:647] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.768353 21245 tablet_copy_client.cc:1002] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.768601 18550 tablet_copy_source_session.cc:205] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.770119 21227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35376539bc944e59b74bfe59d5feb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.776602 21227 tablet_copy_client.cc:783] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.776949 21227 tablet_copy_client.cc:647] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.778241 20932 tablet_copy_client.cc:523] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.780521 21247 tablet_copy_client.cc:1002] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:37.783092 20932 tablet_bootstrap.cc:492] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: Bootstrap starting.
Runtime error: ksck discovered errors
I20240503 14:14:37.796682 20932 log.cc:824] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:37.845052 20949 tablet_copy_client.cc:523] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.848214 20932 tablet_bootstrap.cc:492] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.848850 20932 tablet_bootstrap.cc:492] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: Bootstrap complete.
I20240503 14:14:37.849503 20932 ts_tablet_manager.cc:1367] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: Time spent bootstrapping tablet: real 0.067s user 0.056s sys 0.009s
I20240503 14:14:37.851476 20949 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap starting.
I20240503 14:14:37.856552 20932 raft_consensus.cc:348] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:37.857266 20932 raft_consensus.cc:729] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a632094ba4554a4895d6c0b538b81af8, State: Initialized, Role: LEARNER
I20240503 14:14:37.858413 20932 consensus_queue.cc:260] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:37.865635 20949 log.cc:824] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:37.868636 20932 ts_tablet_manager.cc:1398] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20240503 14:14:37.871263 18420 tablet_copy_service.cc:342] P ea366e6e15a24c52b1ea5a154f942568: Request end of tablet copy session a632094ba4554a4895d6c0b538b81af8-d907591e3c9740e8b5eee42f8d41996b received from {username='slave'} at 127.10.145.7:49165
I20240503 14:14:37.871467 18420 tablet_copy_service.cc:433] P ea366e6e15a24c52b1ea5a154f942568: ending tablet copy session a632094ba4554a4895d6c0b538b81af8-d907591e3c9740e8b5eee42f8d41996b on tablet d907591e3c9740e8b5eee42f8d41996b with peer a632094ba4554a4895d6c0b538b81af8
I20240503 14:14:37.927114 20949 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.927670 20949 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap complete.
I20240503 14:14:37.928124 20949 ts_tablet_manager.cc:1367] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent bootstrapping tablet: real 0.077s user 0.056s sys 0.005s
I20240503 14:14:37.934926 20949 raft_consensus.cc:348] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:37.940840 20949 raft_consensus.cc:729] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Initialized, Role: LEARNER
I20240503 14:14:37.941759 20949 consensus_queue.cc:260] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:37.959578 20949 ts_tablet_manager.cc:1398] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent starting tablet: real 0.031s user 0.010s sys 0.011s
I20240503 14:14:37.960546 18420 tablet_copy_service.cc:342] P ea366e6e15a24c52b1ea5a154f942568: Request end of tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-f0c1a7f5739f4fa59271254e501fb095 received from {username='slave'} at 127.10.145.6:49369
I20240503 14:14:37.960700 18420 tablet_copy_service.cc:433] P ea366e6e15a24c52b1ea5a154f942568: ending tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-f0c1a7f5739f4fa59271254e501fb095 on tablet f0c1a7f5739f4fa59271254e501fb095 with peer 69cbbbedfe03437b8c95b7bb730dbe66
I20240503 14:14:37.963551 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.9:0
--local_ip_for_outbound_sockets=127.10.145.9
--webserver_interface=127.10.145.9
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33209
--builtin_ntp_servers=127.10.145.20:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:14:38.052816 20979 tablet_copy_client.cc:523] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.058063 20979 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: Bootstrap starting.
I20240503 14:14:38.070356 20979 log.cc:824] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:38.120076 20979 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.120823 20979 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: Bootstrap complete.
I20240503 14:14:38.121429 20979 ts_tablet_manager.cc:1367] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: Time spent bootstrapping tablet: real 0.064s user 0.060s sys 0.000s
I20240503 14:14:38.130669 20979 raft_consensus.cc:348] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:38.131342 20979 raft_consensus.cc:729] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Initialized, Role: LEARNER
I20240503 14:14:38.131956 20979 consensus_queue.cc:260] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:38.132611 20475 raft_consensus.cc:1203] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.141914 20979 ts_tablet_manager.cc:1398] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: Time spent starting tablet: real 0.020s user 0.012s sys 0.003s
I20240503 14:14:38.144948 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 80260781ae364d049c2500ff76a06ce8-91743575cf4a4f38b5b43265539237a3 received from {username='slave'} at 127.10.145.8:59415
I20240503 14:14:38.145149 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 80260781ae364d049c2500ff76a06ce8-91743575cf4a4f38b5b43265539237a3 on tablet 91743575cf4a4f38b5b43265539237a3 with peer 80260781ae364d049c2500ff76a06ce8
I20240503 14:14:38.222955 21016 tablet_copy_client.cc:523] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.227017 21016 tablet_bootstrap.cc:492] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: Bootstrap starting.
I20240503 14:14:38.276180 21016 tablet_bootstrap.cc:492] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.276800 21016 tablet_bootstrap.cc:492] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: Bootstrap complete.
I20240503 14:14:38.277119 21016 ts_tablet_manager.cc:1367] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: Time spent bootstrapping tablet: real 0.050s user 0.050s sys 0.000s
I20240503 14:14:38.278327 21016 raft_consensus.cc:348] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:38.278694 21016 raft_consensus.cc:729] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a632094ba4554a4895d6c0b538b81af8, State: Initialized, Role: LEARNER
I20240503 14:14:38.278937 21016 consensus_queue.cc:260] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:38.279552 21016 ts_tablet_manager.cc:1398] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:14:38.280448 18420 tablet_copy_service.cc:342] P ea366e6e15a24c52b1ea5a154f942568: Request end of tablet copy session a632094ba4554a4895d6c0b538b81af8-87fded61bfc240f5846080c328528edc received from {username='slave'} at 127.10.145.7:49165
I20240503 14:14:38.280577 18420 tablet_copy_service.cc:433] P ea366e6e15a24c52b1ea5a154f942568: ending tablet copy session a632094ba4554a4895d6c0b538b81af8-87fded61bfc240f5846080c328528edc on tablet 87fded61bfc240f5846080c328528edc with peer a632094ba4554a4895d6c0b538b81af8
I20240503 14:14:38.282586 21014 tablet_copy_client.cc:523] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.287595 21014 tablet_bootstrap.cc:492] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap starting.
I20240503 14:14:38.295686 21032 tablet_copy_client.cc:523] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.298720 21032 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: Bootstrap starting.
I20240503 14:14:38.301844 21014 log.cc:824] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:38.306208 20342 raft_consensus.cc:1203] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.336623 21032 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.337054 21032 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: Bootstrap complete.
I20240503 14:14:38.337375 21032 ts_tablet_manager.cc:1367] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: Time spent bootstrapping tablet: real 0.039s user 0.033s sys 0.002s
I20240503 14:14:38.338335 21032 raft_consensus.cc:348] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:38.338547 21032 raft_consensus.cc:729] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Initialized, Role: LEARNER
I20240503 14:14:38.338786 21032 consensus_queue.cc:260] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:38.339519 21032 ts_tablet_manager.cc:1398] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:38.340649 21033 tablet_copy_client.cc:523] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.344245 21033 tablet_bootstrap.cc:492] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap starting.
I20240503 14:14:38.346103 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 80260781ae364d049c2500ff76a06ce8-ea3ea2e3d87f4c099abbf5193c239a8b received from {username='slave'} at 127.10.145.8:59415
I20240503 14:14:38.346279 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 80260781ae364d049c2500ff76a06ce8-ea3ea2e3d87f4c099abbf5193c239a8b on tablet ea3ea2e3d87f4c099abbf5193c239a8b with peer 80260781ae364d049c2500ff76a06ce8
I20240503 14:14:38.347282 21039 tablet_copy_client.cc:523] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.347539 20605 raft_consensus.cc:1203] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.351183 21039 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap starting.
I20240503 14:14:38.353020 21014 tablet_bootstrap.cc:492] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.353617 21014 tablet_bootstrap.cc:492] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap complete.
I20240503 14:14:38.354214 21014 ts_tablet_manager.cc:1367] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent bootstrapping tablet: real 0.067s user 0.059s sys 0.009s
I20240503 14:14:38.361069 21014 raft_consensus.cc:348] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:38.361775 21014 raft_consensus.cc:729] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e6a3f22c7ae444e96f6d35cca602e62, State: Initialized, Role: LEARNER
I20240503 14:14:38.362486 21014 consensus_queue.cc:260] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:38.389201 20605 raft_consensus.cc:1203] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.402101 21014 ts_tablet_manager.cc:1398] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent starting tablet: real 0.048s user 0.028s sys 0.017s
I20240503 14:14:38.403470 18420 tablet_copy_service.cc:342] P ea366e6e15a24c52b1ea5a154f942568: Request end of tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-5a23a38ec48148b19bf8102a944ab11e received from {username='slave'} at 127.10.145.5:47647
I20240503 14:14:38.403645 18420 tablet_copy_service.cc:433] P ea366e6e15a24c52b1ea5a154f942568: ending tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-5a23a38ec48148b19bf8102a944ab11e on tablet 5a23a38ec48148b19bf8102a944ab11e with peer 2e6a3f22c7ae444e96f6d35cca602e62
I20240503 14:14:38.444329 21033 tablet_bootstrap.cc:492] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.444872 21033 tablet_bootstrap.cc:492] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap complete.
I20240503 14:14:38.445238 21033 ts_tablet_manager.cc:1367] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent bootstrapping tablet: real 0.101s user 0.047s sys 0.009s
I20240503 14:14:38.446478 21033 raft_consensus.cc:348] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:38.446801 21033 raft_consensus.cc:729] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Initialized, Role: LEARNER
I20240503 14:14:38.447086 21033 consensus_queue.cc:260] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:38.447969 21033 ts_tablet_manager.cc:1398] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:38.449087 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-c18cad62ab6443cea66d08f0980c75b6 received from {username='slave'} at 127.10.145.6:41609
I20240503 14:14:38.449256 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-c18cad62ab6443cea66d08f0980c75b6 on tablet c18cad62ab6443cea66d08f0980c75b6 with peer 69cbbbedfe03437b8c95b7bb730dbe66
I20240503 14:14:38.450232 21039 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.450793 21039 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap complete.
I20240503 14:14:38.452767 21039 ts_tablet_manager.cc:1367] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent bootstrapping tablet: real 0.102s user 0.048s sys 0.007s
I20240503 14:14:38.453796 21039 raft_consensus.cc:348] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:38.454068 21039 raft_consensus.cc:729] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e6a3f22c7ae444e96f6d35cca602e62, State: Initialized, Role: LEARNER
I20240503 14:14:38.454282 21039 consensus_queue.cc:260] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:38.455416 21039 ts_tablet_manager.cc:1398] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:38.456140 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-eeb40a038c4d404d9cb8ea693d9071d7 received from {username='slave'} at 127.10.145.5:45257
I20240503 14:14:38.456293 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-eeb40a038c4d404d9cb8ea693d9071d7 on tablet eeb40a038c4d404d9cb8ea693d9071d7 with peer 2e6a3f22c7ae444e96f6d35cca602e62
I20240503 14:14:38.461544 20210 raft_consensus.cc:1203] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.465418 21066 tablet_copy_client.cc:523] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.469043 21066 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap starting.
I20240503 14:14:38.481524 20475 raft_consensus.cc:1203] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 14:14:38.506640 21280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:38.507195 21280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:38.517632 21280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:38.538730 21280 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:38.538980 21280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:38.539219 21280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.9
I20240503 14:14:38.544601 21066 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.545051 21066 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap complete.
I20240503 14:14:38.545449 21066 ts_tablet_manager.cc:1367] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent bootstrapping tablet: real 0.076s user 0.047s sys 0.003s
I20240503 14:14:38.546346 21066 raft_consensus.cc:348] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:38.546646 21066 raft_consensus.cc:729] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Initialized, Role: LEARNER
I20240503 14:14:38.546926 21066 consensus_queue.cc:260] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:38.548354 21066 ts_tablet_manager.cc:1398] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:38.549163 18420 tablet_copy_service.cc:342] P ea366e6e15a24c52b1ea5a154f942568: Request end of tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-963ecc30d66b43a4ad149f596bd288d9 received from {username='slave'} at 127.10.145.6:49369
I20240503 14:14:38.549325 18420 tablet_copy_service.cc:433] P ea366e6e15a24c52b1ea5a154f942568: ending tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-963ecc30d66b43a4ad149f596bd288d9 on tablet 963ecc30d66b43a4ad149f596bd288d9 with peer 69cbbbedfe03437b8c95b7bb730dbe66
I20240503 14:14:38.552654 21280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.9:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.10.145.9
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.10.145.62:33209
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.9
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:38.554466 21280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:38.556557 21280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:38.573349 21353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:38.575209 21083 tablet_copy_client.cc:523] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.579530 21083 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: Bootstrap starting.
W20240503 14:14:38.582991 21354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:38.604503 21099 tablet_copy_client.cc:523] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 14:14:38.604668 21358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:38.608633 21099 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: Bootstrap starting.
I20240503 14:14:38.616139 21280 server_base.cc:1034] running on GCE node
I20240503 14:14:38.617100 21280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:38.631409 21280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:38.635720 21110 tablet_copy_client.cc:523] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.639427 21110 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: Bootstrap starting.
I20240503 14:14:38.642081 20342 raft_consensus.cc:1203] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.647668 21280 hybrid_clock.cc:648] HybridClock initialized: now 1714745678647523 us; error 124 us; skew 500 ppm
I20240503 14:14:38.648164 21280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:38.655524 21280 webserver.cc:460] Webserver started at http://127.10.145.9:41259/ using document root <none> and password file <none>
I20240503 14:14:38.656061 21083 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.656574 21083 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: Bootstrap complete.
I20240503 14:14:38.656872 21083 ts_tablet_manager.cc:1367] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: Time spent bootstrapping tablet: real 0.077s user 0.045s sys 0.008s
I20240503 14:14:38.657094 21280 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:38.657301 21280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:38.657783 21280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:38.657898 21083 raft_consensus.cc:348] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:38.658203 21083 raft_consensus.cc:729] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a632094ba4554a4895d6c0b538b81af8, State: Initialized, Role: LEARNER
I20240503 14:14:38.658504 21083 consensus_queue.cc:260] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:38.662137 21083 ts_tablet_manager.cc:1398] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:14:38.664546 21280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "c36ba962bcb34dbea55f4e55689bd38a"
format_stamp: "Formatted at 2024-05-03 14:14:38 on dist-test-slave-v549"
I20240503 14:14:38.666790 18550 tablet_copy_service.cc:342] P 55339e38e7aa4bb7b7b42aa206108722: Request end of tablet copy session a632094ba4554a4895d6c0b538b81af8-7c63b35eef494f0e8928c4ca0f74ec32 received from {username='slave'} at 127.10.145.7:40207
I20240503 14:14:38.666962 18550 tablet_copy_service.cc:433] P 55339e38e7aa4bb7b7b42aa206108722: ending tablet copy session a632094ba4554a4895d6c0b538b81af8-7c63b35eef494f0e8928c4ca0f74ec32 on tablet 7c63b35eef494f0e8928c4ca0f74ec32 with peer a632094ba4554a4895d6c0b538b81af8
I20240503 14:14:38.677464 21280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "c36ba962bcb34dbea55f4e55689bd38a"
format_stamp: "Formatted at 2024-05-03 14:14:38 on dist-test-slave-v549"
I20240503 14:14:38.685963 21099 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.686463 21099 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: Bootstrap complete.
I20240503 14:14:38.686756 21099 ts_tablet_manager.cc:1367] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: Time spent bootstrapping tablet: real 0.078s user 0.045s sys 0.007s
I20240503 14:14:38.687597 21099 raft_consensus.cc:348] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:38.687894 21099 raft_consensus.cc:729] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Initialized, Role: LEARNER
I20240503 14:14:38.688107 21099 consensus_queue.cc:260] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:38.689878 21099 ts_tablet_manager.cc:1398] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:38.690757 18680 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session 80260781ae364d049c2500ff76a06ce8-7570d5bce0e5481d8dded653617be5a7 received from {username='slave'} at 127.10.145.8:39351
I20240503 14:14:38.690950 18680 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session 80260781ae364d049c2500ff76a06ce8-7570d5bce0e5481d8dded653617be5a7 on tablet 7570d5bce0e5481d8dded653617be5a7 with peer 80260781ae364d049c2500ff76a06ce8
I20240503 14:14:38.699162 21110 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.699656 21110 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: Bootstrap complete.
I20240503 14:14:38.699962 21110 ts_tablet_manager.cc:1367] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: Time spent bootstrapping tablet: real 0.061s user 0.036s sys 0.009s
I20240503 14:14:38.700824 21110 raft_consensus.cc:348] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:38.701102 21110 raft_consensus.cc:729] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a632094ba4554a4895d6c0b538b81af8, State: Initialized, Role: LEARNER
I20240503 14:14:38.701342 21110 consensus_queue.cc:260] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:38.702607 21110 ts_tablet_manager.cc:1398] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20240503 14:14:38.703290 18550 tablet_copy_service.cc:342] P 55339e38e7aa4bb7b7b42aa206108722: Request end of tablet copy session a632094ba4554a4895d6c0b538b81af8-7f9705f607a94835ad7e6c20668b80ad received from {username='slave'} at 127.10.145.7:40207
I20240503 14:14:38.703419 18550 tablet_copy_service.cc:433] P 55339e38e7aa4bb7b7b42aa206108722: ending tablet copy session a632094ba4554a4895d6c0b538b81af8-7f9705f607a94835ad7e6c20668b80ad on tablet 7f9705f607a94835ad7e6c20668b80ad with peer a632094ba4554a4895d6c0b538b81af8
I20240503 14:14:38.710633 21280 fs_manager.cc:691] Time spent creating directory manager: real 0.032s user 0.017s sys 0.000s
I20240503 14:14:38.716425 21383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.718806 21280 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:14:38.719012 21280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "c36ba962bcb34dbea55f4e55689bd38a"
format_stamp: "Formatted at 2024-05-03 14:14:38 on dist-test-slave-v549"
I20240503 14:14:38.719285 21280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:38.745671 20875 raft_consensus.cc:1050] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: attempting to promote NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 to VOTER
I20240503 14:14:38.746799 20875 consensus_queue.cc:237] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:38.750517 18528 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.750653 18267 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.751031 21148 tablet_copy_client.cc:523] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.751570 20870 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.752662 21280 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:38.753801 21280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:38.754107 21280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:38.754732 21148 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap starting.
I20240503 14:14:38.758752 20392 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.760815 20342 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.760805 21280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:14:38.762879 21399 raft_consensus.cc:1050] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568: attempting to promote NON_VOTER a632094ba4554a4895d6c0b538b81af8 to VOTER
I20240503 14:14:38.763772 21399 consensus_queue.cc:237] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:38.765260 20875 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.772977 20392 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:38.778471 21280 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:14:38.778525 18528 raft_consensus.cc:1261] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.778626 21280 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.778741 21280 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:14:38.778820 21280 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:38.779345 18268 raft_consensus.cc:1261] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.780145 20475 raft_consensus.cc:1261] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.780534 21399 consensus_queue.cc:1035] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.780967 20875 consensus_queue.cc:1035] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.781598 20717 raft_consensus.cc:1050] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 80260781ae364d049c2500ff76a06ce8 to VOTER
I20240503 14:14:38.781805 20875 consensus_queue.cc:1035] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.782737 20717 consensus_queue.cc:237] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:38.797317 21399 raft_consensus.cc:2932] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:38.800720 20845 raft_consensus.cc:1050] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 80260781ae364d049c2500ff76a06ce8 to VOTER
I20240503 14:14:38.801735 20845 consensus_queue.cc:237] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:38.802186 18528 raft_consensus.cc:2932] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:38.806174 20605 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.808938 18659 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.812098 21042 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.811867 18529 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:38.812455 18400 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.812880 21042 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.813575 21414 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.818847 21167 tablet_copy_client.cc:523] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.822245 20475 raft_consensus.cc:1203] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.822827 18267 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:38.827425 21167 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: Bootstrap starting.
I20240503 14:14:38.832237 18269 raft_consensus.cc:2932] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:38.835592 18121 catalog_manager.cc:5484] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:38.837986 18119 catalog_manager.cc:5484] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:38.841593 20475 raft_consensus.cc:2932] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:38.845165 18528 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.845341 20604 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.845942 21042 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.846410 18658 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.847791 21042 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.848356 21042 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.849439 21182 tablet_copy_client.cc:523] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.857962 21182 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap starting.
I20240503 14:14:38.866400 20341 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:38.896215 20342 raft_consensus.cc:1203] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.907099 21420 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:38.914631 18399 consensus_queue.cc:237] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:38.918361 18267 raft_consensus.cc:1261] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:38.918603 20475 raft_consensus.cc:1261] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:38.919344 18658 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:38.921074 20875 consensus_queue.cc:1035] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:38.923473 20392 consensus_queue.cc:1035] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:38.924211 20870 raft_consensus.cc:1050] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568: attempting to promote NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 to VOTER
I20240503 14:14:38.924664 20717 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:38.925083 20870 consensus_queue.cc:237] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:38.925340 21399 raft_consensus.cc:2932] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:38.929015 18120 catalog_manager.cc:5484] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:38.928716 20603 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:38.930501 18267 raft_consensus.cc:2932] T d907591e3c9740e8b5eee42f8d41996b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:38.922690 20604 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:38.932477 18657 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:38.937162 18114 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d907591e3c9740e8b5eee42f8d41996b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:38.939517 20210 raft_consensus.cc:1203] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.939559 18128 catalog_manager.cc:5484] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:38.939994 20475 raft_consensus.cc:2932] T d907591e3c9740e8b5eee42f8d41996b P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:38.941602 18119 catalog_manager.cc:5484] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:38.949784 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet d907591e3c9740e8b5eee42f8d41996b with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:38.950362 18529 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:38.956339 18269 raft_consensus.cc:1261] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.956511 20210 raft_consensus.cc:1261] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.956990 21399 consensus_queue.cc:1035] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.957391 20392 consensus_queue.cc:1035] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.974730 21443 tablet_replica.cc:330] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:38.977306 21443 raft_consensus.cc:2229] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:38.978729 21443 raft_consensus.cc:2258] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:38.979857 20392 raft_consensus.cc:2932] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:38.981812 21443 ts_tablet_manager.cc:1875] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:38.984030 18269 raft_consensus.cc:2932] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:38.986027 20210 raft_consensus.cc:2932] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:38.993363 18658 raft_consensus.cc:1261] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:38.993973 21399 consensus_queue.cc:1035] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:38.994321 18128 catalog_manager.cc:5484] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62 reported cstate change: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.011814 21443 ts_tablet_manager.cc:1888] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:39.014771 18399 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:39.018244 18398 consensus_queue.cc:237] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:39.021963 21399 raft_consensus.cc:1050] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568: attempting to promote NON_VOTER a632094ba4554a4895d6c0b538b81af8 to VOTER
I20240503 14:14:39.022677 21443 log.cc:1196] T d907591e3c9740e8b5eee42f8d41996b P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/d907591e3c9740e8b5eee42f8d41996b
I20240503 14:14:39.022912 21399 consensus_queue.cc:237] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:39.024755 20210 raft_consensus.cc:1261] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.024945 18658 raft_consensus.cc:2932] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.025478 20392 consensus_queue.cc:1035] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.027077 18269 raft_consensus.cc:1261] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.028028 20392 consensus_queue.cc:1035] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.029234 18529 raft_consensus.cc:1261] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.029306 20475 raft_consensus.cc:1261] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.029582 18267 raft_consensus.cc:1261] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.030009 21446 consensus_queue.cc:1035] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.030717 21446 consensus_queue.cc:1035] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.031638 20392 consensus_queue.cc:1035] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.034277 20210 raft_consensus.cc:2932] T 5a23a38ec48148b19bf8102a944ab11e P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.035542 18269 raft_consensus.cc:2932] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.037326 20392 raft_consensus.cc:2932] T 5a23a38ec48148b19bf8102a944ab11e P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.042781 18114 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a23a38ec48148b19bf8102a944ab11e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:39.045941 18128 catalog_manager.cc:5484] T 5a23a38ec48148b19bf8102a944ab11e P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.050462 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a23a38ec48148b19bf8102a944ab11e with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:39.058349 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet d907591e3c9740e8b5eee42f8d41996b (table rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]) successfully deleted
I20240503 14:14:39.061604 20604 raft_consensus.cc:1203] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.076149 21460 tablet_replica.cc:330] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:39.076592 21399 raft_consensus.cc:2932] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.082051 18530 raft_consensus.cc:2932] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.082526 20475 raft_consensus.cc:2932] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.083213 18267 raft_consensus.cc:2932] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.084558 21460 raft_consensus.cc:2229] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:39.085747 21460 raft_consensus.cc:2258] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:39.087901 18120 catalog_manager.cc:5484] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:39.100505 18399 consensus_queue.cc:237] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:39.105811 20845 raft_consensus.cc:1050] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 to VOTER
I20240503 14:14:39.106689 20845 consensus_queue.cc:237] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:39.107427 21460 ts_tablet_manager.cc:1875] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:39.109083 18267 raft_consensus.cc:1261] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.109745 20392 consensus_queue.cc:1035] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.112725 20875 raft_consensus.cc:2932] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.113924 18267 raft_consensus.cc:2932] T 87fded61bfc240f5846080c328528edc P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.117074 18114 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87fded61bfc240f5846080c328528edc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:39.119149 18121 catalog_manager.cc:5484] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:39.123517 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet 87fded61bfc240f5846080c328528edc with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:39.123943 21443 tablet_replica.cc:330] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:39.124121 20474 raft_consensus.cc:1261] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.124162 21443 raft_consensus.cc:2229] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:39.124402 20342 raft_consensus.cc:1261] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.124507 21443 raft_consensus.cc:2258] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:39.125105 20717 consensus_queue.cc:1035] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.126063 21399 consensus_queue.cc:1035] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.126288 21443 ts_tablet_manager.cc:1875] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:39.127853 20474 raft_consensus.cc:2932] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.131242 18399 raft_consensus.cc:1261] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.131668 18530 raft_consensus.cc:1261] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.132263 21420 consensus_queue.cc:1035] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.133113 21420 consensus_queue.cc:1035] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.133936 21443 ts_tablet_manager.cc:1888] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:39.134128 21443 log.cc:1196] T 87fded61bfc240f5846080c328528edc P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/87fded61bfc240f5846080c328528edc
I20240503 14:14:39.134945 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet 87fded61bfc240f5846080c328528edc (table rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]) successfully deleted
I20240503 14:14:39.136139 20475 raft_consensus.cc:1203] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.142467 21420 raft_consensus.cc:2932] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.145726 18399 raft_consensus.cc:2932] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.146107 18529 raft_consensus.cc:2932] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.152734 18121 catalog_manager.cc:5484] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.157562 21182 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:39.158068 21182 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap complete.
I20240503 14:14:39.158389 21182 ts_tablet_manager.cc:1367] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent bootstrapping tablet: real 0.301s user 0.049s sys 0.001s
I20240503 14:14:39.159260 21182 raft_consensus.cc:348] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:39.159545 21182 raft_consensus.cc:729] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Initialized, Role: LEARNER
I20240503 14:14:39.159766 21182 consensus_queue.cc:260] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:39.160781 18267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea3ea2e3d87f4c099abbf5193c239a8b"
dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58212
I20240503 14:14:39.161065 21182 ts_tablet_manager.cc:1398] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:39.161056 18267 raft_consensus.cc:595] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:39.161895 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-a39fbf1d8f6f46ccab3c2b992ed0c63e received from {username='slave'} at 127.10.145.6:41609
I20240503 14:14:39.162037 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-a39fbf1d8f6f46ccab3c2b992ed0c63e on tablet a39fbf1d8f6f46ccab3c2b992ed0c63e with peer 69cbbbedfe03437b8c95b7bb730dbe66
I20240503 14:14:39.168537 21460 ts_tablet_manager.cc:1888] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:39.169672 20342 raft_consensus.cc:2932] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.173712 21460 log.cc:1196] T 5a23a38ec48148b19bf8102a944ab11e P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/5a23a38ec48148b19bf8102a944ab11e
I20240503 14:14:39.179636 18267 consensus_queue.cc:237] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:39.181222 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet 5a23a38ec48148b19bf8102a944ab11e (table rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]) successfully deleted
I20240503 14:14:39.184895 18529 raft_consensus.cc:1261] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.185606 21042 consensus_queue.cc:1035] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.185930 20341 raft_consensus.cc:1261] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.187721 21042 consensus_queue.cc:1035] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.190460 21414 raft_consensus.cc:2932] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.194957 21211 tablet_copy_client.cc:523] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:39.196707 20341 raft_consensus.cc:2932] T c18cad62ab6443cea66d08f0980c75b6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.200219 21211 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap starting.
I20240503 14:14:39.203802 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c18cad62ab6443cea66d08f0980c75b6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:39.206827 18528 raft_consensus.cc:2932] T c18cad62ab6443cea66d08f0980c75b6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.214812 18121 catalog_manager.cc:5484] T c18cad62ab6443cea66d08f0980c75b6 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:39.220507 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet c18cad62ab6443cea66d08f0980c75b6 with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:39.221211 21167 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:39.221657 21480 tablet_replica.cc:330] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:39.222265 21480 raft_consensus.cc:2229] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:39.222988 21167 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: Bootstrap complete.
I20240503 14:14:39.223474 21167 ts_tablet_manager.cc:1367] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: Time spent bootstrapping tablet: real 0.396s user 0.049s sys 0.007s
I20240503 14:14:39.223887 21480 raft_consensus.cc:2258] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:39.224349 21167 raft_consensus.cc:348] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:39.224637 21167 raft_consensus.cc:729] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Initialized, Role: LEARNER
I20240503 14:14:39.224869 21167 consensus_queue.cc:260] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:39.222661 21148 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:39.225603 21148 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap complete.
I20240503 14:14:39.225931 21148 ts_tablet_manager.cc:1367] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent bootstrapping tablet: real 0.471s user 0.055s sys 0.001s
I20240503 14:14:39.226820 21148 raft_consensus.cc:348] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:39.227140 21148 raft_consensus.cc:729] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e6a3f22c7ae444e96f6d35cca602e62, State: Initialized, Role: LEARNER
I20240503 14:14:39.227380 21148 consensus_queue.cc:260] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:39.228521 21480 ts_tablet_manager.cc:1875] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:39.229089 21167 ts_tablet_manager.cc:1398] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:14:39.230038 21148 ts_tablet_manager.cc:1398] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240503 14:14:39.230803 18420 tablet_copy_service.cc:342] P ea366e6e15a24c52b1ea5a154f942568: Request end of tablet copy session 80260781ae364d049c2500ff76a06ce8-8c337ef0405e4856bd2a608a6a602a49 received from {username='slave'} at 127.10.145.8:41697
I20240503 14:14:39.230935 18420 tablet_copy_service.cc:433] P ea366e6e15a24c52b1ea5a154f942568: ending tablet copy session 80260781ae364d049c2500ff76a06ce8-8c337ef0405e4856bd2a608a6a602a49 on tablet 8c337ef0405e4856bd2a608a6a602a49 with peer 80260781ae364d049c2500ff76a06ce8
I20240503 14:14:39.230966 18680 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-9577e6d3801f42f89afe8a48a77031dd received from {username='slave'} at 127.10.145.5:43483
I20240503 14:14:39.231137 18680 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-9577e6d3801f42f89afe8a48a77031dd on tablet 9577e6d3801f42f89afe8a48a77031dd with peer 2e6a3f22c7ae444e96f6d35cca602e62
I20240503 14:14:39.237449 21215 tablet_copy_client.cc:523] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:39.241232 21215 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap starting.
I20240503 14:14:39.245196 21480 ts_tablet_manager.cc:1888] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:39.247274 21480 log.cc:1196] T c18cad62ab6443cea66d08f0980c75b6 P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/c18cad62ab6443cea66d08f0980c75b6
I20240503 14:14:39.249725 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet c18cad62ab6443cea66d08f0980c75b6 (table rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]) successfully deleted
I20240503 14:14:39.257193 21485 raft_consensus.cc:1050] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722: attempting to promote NON_VOTER a632094ba4554a4895d6c0b538b81af8 to VOTER
I20240503 14:14:39.259809 21485 consensus_queue.cc:237] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:39.262573 21217 tablet_copy_client.cc:523] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:39.266346 21217 tablet_bootstrap.cc:492] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: Bootstrap starting.
I20240503 14:14:39.268865 21235 tablet_copy_client.cc:523] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:39.273919 21235 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap starting.
I20240503 14:14:39.280946 21227 tablet_copy_client.cc:523] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:39.284968 21227 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: Bootstrap starting.
I20240503 14:14:39.287492 21211 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:39.288035 21211 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap complete.
I20240503 14:14:39.288353 21211 ts_tablet_manager.cc:1367] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent bootstrapping tablet: real 0.088s user 0.043s sys 0.012s
I20240503 14:14:39.289304 21211 raft_consensus.cc:348] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:39.289574 21211 raft_consensus.cc:729] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e6a3f22c7ae444e96f6d35cca602e62, State: Initialized, Role: LEARNER
I20240503 14:14:39.289822 21211 consensus_queue.cc:260] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:39.291209 21211 ts_tablet_manager.cc:1398] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:14:39.291992 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-79373bd79a934eaab67bbcf2208e7b5b received from {username='slave'} at 127.10.145.5:45257
I20240503 14:14:39.292176 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-79373bd79a934eaab67bbcf2208e7b5b on tablet 79373bd79a934eaab67bbcf2208e7b5b with peer 2e6a3f22c7ae444e96f6d35cca602e62
I20240503 14:14:39.305225 18657 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.308542 20475 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.309764 21482 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.314493 18267 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.318011 21361 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.318882 21361 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.336946 20875 raft_consensus.cc:1050] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568: attempting to promote NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 to VOTER
I20240503 14:14:39.338042 20875 consensus_queue.cc:237] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:39.346205 20342 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.346662 18657 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.347023 21399 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.349496 20392 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.355821 18528 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.357350 21399 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.358577 20210 raft_consensus.cc:1203] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.366526 20341 raft_consensus.cc:1203] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.386252 21215 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:39.388120 21227 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:39.388706 21227 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: Bootstrap complete.
I20240503 14:14:39.388947 21215 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap complete.
I20240503 14:14:39.389143 21227 ts_tablet_manager.cc:1367] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: Time spent bootstrapping tablet: real 0.104s user 0.047s sys 0.005s
I20240503 14:14:39.389969 21215 ts_tablet_manager.cc:1367] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent bootstrapping tablet: real 0.149s user 0.049s sys 0.007s
I20240503 14:14:39.390143 21227 raft_consensus.cc:348] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:39.392016 21215 raft_consensus.cc:348] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:39.392463 21235 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:39.392959 21235 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap complete.
I20240503 14:14:39.393337 21227 raft_consensus.cc:729] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Initialized, Role: LEARNER
I20240503 14:14:39.393694 21227 consensus_queue.cc:260] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:39.393954 21215 raft_consensus.cc:729] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Initialized, Role: LEARNER
I20240503 14:14:39.394629 21215 consensus_queue.cc:260] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:39.397148 21215 ts_tablet_manager.cc:1398] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:14:39.397827 21227 ts_tablet_manager.cc:1398] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 14:14:39.399402 21235 ts_tablet_manager.cc:1367] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent bootstrapping tablet: real 0.126s user 0.054s sys 0.004s
I20240503 14:14:39.399585 18550 tablet_copy_service.cc:342] P 55339e38e7aa4bb7b7b42aa206108722: Request end of tablet copy session 80260781ae364d049c2500ff76a06ce8-c35376539bc944e59b74bfe59d5feb9e received from {username='slave'} at 127.10.145.8:40531
I20240503 14:14:39.399734 18550 tablet_copy_service.cc:433] P 55339e38e7aa4bb7b7b42aa206108722: ending tablet copy session 80260781ae364d049c2500ff76a06ce8-c35376539bc944e59b74bfe59d5feb9e on tablet c35376539bc944e59b74bfe59d5feb9e with peer 80260781ae364d049c2500ff76a06ce8
I20240503 14:14:39.400444 21235 raft_consensus.cc:348] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:39.400754 21217 tablet_bootstrap.cc:492] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:39.401199 21217 tablet_bootstrap.cc:492] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: Bootstrap complete.
I20240503 14:14:39.402426 21235 raft_consensus.cc:729] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e6a3f22c7ae444e96f6d35cca602e62, State: Initialized, Role: LEARNER
I20240503 14:14:39.402808 21235 consensus_queue.cc:260] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:39.404121 18550 tablet_copy_service.cc:342] P 55339e38e7aa4bb7b7b42aa206108722: Request end of tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-19b1b6be3bda47829f53765282eae9f9 received from {username='slave'} at 127.10.145.6:38759
I20240503 14:14:39.404260 18550 tablet_copy_service.cc:433] P 55339e38e7aa4bb7b7b42aa206108722: ending tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-19b1b6be3bda47829f53765282eae9f9 on tablet 19b1b6be3bda47829f53765282eae9f9 with peer 69cbbbedfe03437b8c95b7bb730dbe66
I20240503 14:14:39.407555 21235 ts_tablet_manager.cc:1398] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 14:14:39.408833 21217 ts_tablet_manager.cc:1367] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: Time spent bootstrapping tablet: real 0.143s user 0.035s sys 0.013s
I20240503 14:14:39.410219 21217 raft_consensus.cc:348] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:39.410362 21361 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.411899 21217 raft_consensus.cc:729] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a632094ba4554a4895d6c0b538b81af8, State: Initialized, Role: LEARNER
I20240503 14:14:39.412392 21217 consensus_queue.cc:260] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:39.413796 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-eedc3cfb2dff4f9084446d89b9b3c0f4 received from {username='slave'} at 127.10.145.5:45257
I20240503 14:14:39.413947 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-eedc3cfb2dff4f9084446d89b9b3c0f4 on tablet eedc3cfb2dff4f9084446d89b9b3c0f4 with peer 2e6a3f22c7ae444e96f6d35cca602e62
I20240503 14:14:39.414054 21217 ts_tablet_manager.cc:1398] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:14:39.414762 18680 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session a632094ba4554a4895d6c0b538b81af8-4d24e31ff3ad41d98214bb41d8c28fda received from {username='slave'} at 127.10.145.7:47505
I20240503 14:14:39.414906 18680 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session a632094ba4554a4895d6c0b538b81af8-4d24e31ff3ad41d98214bb41d8c28fda on tablet 4d24e31ff3ad41d98214bb41d8c28fda with peer a632094ba4554a4895d6c0b538b81af8
I20240503 14:14:39.416566 20474 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.418126 18658 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.419381 18121 catalog_manager.cc:5484] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:39.427417 21399 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.430176 18658 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.434684 20342 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.436873 18530 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.441543 18121 catalog_manager.cc:5484] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.444305 18269 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.452459 21420 raft_consensus.cc:979] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: : Instructing follower 80260781ae364d049c2500ff76a06ce8 to start an election
I20240503 14:14:39.452667 21420 raft_consensus.cc:1067] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Signalling peer 80260781ae364d049c2500ff76a06ce8 to start an election
I20240503 14:14:39.453006 18530 consensus_queue.cc:237] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:39.454612 20210 raft_consensus.cc:1203] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.454989 20604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea3ea2e3d87f4c099abbf5193c239a8b"
dest_uuid: "80260781ae364d049c2500ff76a06ce8"
from {username='slave'} at 127.10.145.1:56471
I20240503 14:14:39.455317 20604 raft_consensus.cc:482] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:39.455498 20604 raft_consensus.cc:3037] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:39.459313 20604 raft_consensus.cc:504] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:39.461191 20604 leader_election.cc:288] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [CANDIDATE]: Term 2 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:39.461655 18657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ea2e3d87f4c099abbf5193c239a8b" candidate_uuid: "80260781ae364d049c2500ff76a06ce8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:39.461921 18657 raft_consensus.cc:3037] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:39.461937 20342 raft_consensus.cc:1203] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.462103 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ea2e3d87f4c099abbf5193c239a8b" candidate_uuid: "80260781ae364d049c2500ff76a06ce8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:39.462294 18528 raft_consensus.cc:3037] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:39.465204 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3ea2e3d87f4c099abbf5193c239a8b" candidate_uuid: "80260781ae364d049c2500ff76a06ce8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:39.465458 18269 raft_consensus.cc:3032] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:39.465555 18269 raft_consensus.cc:729] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:39.465900 18269 consensus_queue.cc:260] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:39.466389 18657 raft_consensus.cc:2454] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80260781ae364d049c2500ff76a06ce8 in term 2.
I20240503 14:14:39.466428 18269 raft_consensus.cc:3037] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:39.470490 20475 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.471068 21361 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.473049 18528 raft_consensus.cc:2454] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80260781ae364d049c2500ff76a06ce8 in term 2.
I20240503 14:14:39.473518 20541 leader_election.cc:302] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722, 80260781ae364d049c2500ff76a06ce8; no voters:
I20240503 14:14:39.473980 21316 raft_consensus.cc:2781] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:39.473858 21482 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.474828 21420 raft_consensus.cc:1050] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 to VOTER
I20240503 14:14:39.475658 21420 consensus_queue.cc:237] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:39.478245 20474 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.481889 18267 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.485013 18119 catalog_manager.cc:5484] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 reported cstate change: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.486722 18269 raft_consensus.cc:2454] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80260781ae364d049c2500ff76a06ce8 in term 2.
I20240503 14:14:39.489471 20210 raft_consensus.cc:1203] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.490042 18398 consensus_queue.cc:237] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:39.491339 21361 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.493563 20342 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.493860 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9705f607a94835ad7e6c20668b80ad with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:39.493978 18658 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.494263 21460 tablet_replica.cc:330] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:39.494236 20392 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.494475 21460 raft_consensus.cc:2229] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:39.494596 20875 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.494720 21460 raft_consensus.cc:2258] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:39.496042 21460 ts_tablet_manager.cc:1875] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:39.496762 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7f9705f607a94835ad7e6c20668b80ad with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:39.497537 21316 raft_consensus.cc:686] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 2 LEADER]: Becoming Leader. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Running, Role: LEADER
I20240503 14:14:39.498095 21316 consensus_queue.cc:237] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:39.502009 18269 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.502151 18121 catalog_manager.cc:5484] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 reported cstate change: term changed from 1 to 2, leader changed from 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) to 80260781ae364d049c2500ff76a06ce8 (127.10.145.8). New cstate: current_term: 2 leader_uuid: "80260781ae364d049c2500ff76a06ce8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:39.506412 20392 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.508118 21460 ts_tablet_manager.cc:1888] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:39.508251 21460 log.cc:1196] T 7f9705f607a94835ad7e6c20668b80ad P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/7f9705f607a94835ad7e6c20668b80ad
I20240503 14:14:39.509135 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet 7f9705f607a94835ad7e6c20668b80ad (table rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]) successfully deleted
I20240503 14:14:39.510707 20342 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.514623 18114 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 963ecc30d66b43a4ad149f596bd288d9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:39.517048 18120 catalog_manager.cc:5484] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:39.520270 18658 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:39.522243 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet 963ecc30d66b43a4ad149f596bd288d9 with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:39.522625 21443 tablet_replica.cc:330] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:39.522840 21443 raft_consensus.cc:2229] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:39.523167 21443 raft_consensus.cc:2258] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:39.524312 21443 ts_tablet_manager.cc:1875] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:39.530947 21485 raft_consensus.cc:1050] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722: attempting to promote NON_VOTER a632094ba4554a4895d6c0b538b81af8 to VOTER
I20240503 14:14:39.531009 21443 ts_tablet_manager.cc:1888] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:39.531275 21443 log.cc:1196] T 963ecc30d66b43a4ad149f596bd288d9 P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/963ecc30d66b43a4ad149f596bd288d9
I20240503 14:14:39.531917 21485 consensus_queue.cc:237] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:39.533674 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet 963ecc30d66b43a4ad149f596bd288d9 (table rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]) successfully deleted
I20240503 14:14:39.535809 18398 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.536229 20475 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.537544 20604 raft_consensus.cc:1203] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.540884 21482 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.541529 21482 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.545186 18658 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.545923 21485 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.548141 20210 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.548971 21414 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.550482 18657 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.553560 21361 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.556913 18399 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.557816 21414 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.560837 20475 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.561897 20474 raft_consensus.cc:1203] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.561913 18120 catalog_manager.cc:5484] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:39.568642 18528 consensus_queue.cc:237] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:39.570693 18659 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.571557 21420 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.578742 18659 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.580763 21485 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.581447 20788 raft_consensus.cc:1050] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER 80260781ae364d049c2500ff76a06ce8 to VOTER
I20240503 14:14:39.582634 20788 consensus_queue.cc:237] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:39.584659 21478 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.588349 18657 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.591850 20604 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.592756 18267 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.593508 20918 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.593788 18528 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.594224 20918 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.594725 20918 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.594821 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c63b35eef494f0e8928c4ca0f74ec32 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:39.601687 20845 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.605785 20210 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.607275 18399 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.611413 18660 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.613325 18120 catalog_manager.cc:5484] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.616461 20474 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.618080 21485 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.621999 18119 catalog_manager.cc:5484] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:39.627552 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c63b35eef494f0e8928c4ca0f74ec32 with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:39.627969 21480 tablet_replica.cc:330] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:39.628208 21480 raft_consensus.cc:2229] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:39.628512 21480 pending_rounds.cc:62] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: Trying to abort 1 pending ops.
I20240503 14:14:39.628633 21480 pending_rounds.cc:68] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023598303359778816 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7c63b35eef494f0e8928c4ca0f74ec32" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } } }
I20240503 14:14:39.629068 21480 raft_consensus.cc:2894] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 14:14:39.629209 21480 raft_consensus.cc:2866] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:14:39.629343 21480 raft_consensus.cc:2258] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:39.630911 21480 ts_tablet_manager.cc:1875] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:39.631417 20475 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:39.642036 18269 consensus_queue.cc:237] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:39.648566 20210 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.649055 18398 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.649662 20717 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.651007 20717 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.653898 20717 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.654742 18398 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.659986 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eeb40a038c4d404d9cb8ea693d9071d7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:39.660598 20210 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.661026 18119 catalog_manager.cc:5484] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.664666 20788 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:39.669785 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb40a038c4d404d9cb8ea693d9071d7 with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:39.670323 21460 tablet_replica.cc:330] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:39.670570 21460 raft_consensus.cc:2229] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:39.670977 21460 raft_consensus.cc:2258] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:39.673007 21480 ts_tablet_manager.cc:1888] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:39.673272 21480 log.cc:1196] T 7c63b35eef494f0e8928c4ca0f74ec32 P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/7c63b35eef494f0e8928c4ca0f74ec32
I20240503 14:14:39.672894 18529 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:39.677379 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet 7c63b35eef494f0e8928c4ca0f74ec32 (table rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]) successfully deleted
I20240503 14:14:39.679204 21460 ts_tablet_manager.cc:1875] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:39.682232 18269 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:39.687289 18119 catalog_manager.cc:5484] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:39.704298 20604 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:39.705657 18657 consensus_queue.cc:237] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:39.706270 21460 ts_tablet_manager.cc:1888] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:39.706460 21460 log.cc:1196] T eeb40a038c4d404d9cb8ea693d9071d7 P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/eeb40a038c4d404d9cb8ea693d9071d7
I20240503 14:14:39.710618 18530 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.712019 20788 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.714987 21546 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:39.716089 18529 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:39.717744 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet eeb40a038c4d404d9cb8ea693d9071d7 (table rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]) successfully deleted
I20240503 14:14:39.722223 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7570d5bce0e5481d8dded653617be5a7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:39.722939 18119 catalog_manager.cc:5484] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:39.729209 20603 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.730346 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet 7570d5bce0e5481d8dded653617be5a7 with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:39.730366 20788 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.731364 20603 raft_consensus.cc:1203] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.732093 20605 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:39.746188 21554 tablet_replica.cc:330] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:39.747449 21554 raft_consensus.cc:2229] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:39.748090 21554 raft_consensus.cc:2258] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:39.750487 21554 ts_tablet_manager.cc:1875] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:39.759243 21554 ts_tablet_manager.cc:1888] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:39.759567 21554 log.cc:1196] T 7570d5bce0e5481d8dded653617be5a7 P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/7570d5bce0e5481d8dded653617be5a7
I20240503 14:14:39.760761 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet 7570d5bce0e5481d8dded653617be5a7 (table rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]) successfully deleted
I20240503 14:14:39.828130 21280 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.9:45329
I20240503 14:14:39.828153 21632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.9:45329 every 8 connection(s)
I20240503 14:14:39.829933 21280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 14:14:39.836236 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 21280
I20240503 14:14:39.836490 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240503 14:14:39.853055 21635 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33209
I20240503 14:14:39.853478 21635 heartbeater.cc:464] Registering TS with master...
I20240503 14:14:39.854568 21635 heartbeater.cc:511] Master 127.10.145.62:33209 requested a full tablet report, sending...
I20240503 14:14:39.856109 18119 ts_manager.cc:194] Registered new tserver with Master: c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329)
I20240503 14:14:39.857182 18119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.9:36467
I20240503 14:14:39.858980 18267 consensus_queue.cc:237] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:39.861455 18657 raft_consensus.cc:1261] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:39.861816 18203 consensus_peers.cc:485] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 -> Peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Couldn't send request to peer 69cbbbedfe03437b8c95b7bb730dbe66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10a4ca83469d4dab823a26d6cbe952a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:39.862156 20845 consensus_queue.cc:1035] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:39.862259 18528 raft_consensus.cc:1261] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:39.862811 20845 consensus_queue.cc:1035] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:39.865723 21042 raft_consensus.cc:2932] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:39.866566 18657 raft_consensus.cc:2932] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:39.869908 18528 raft_consensus.cc:2932] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:39.871549 18120 catalog_manager.cc:5484] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:39.880640 20788 raft_consensus.cc:1050] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 to VOTER
I20240503 14:14:39.881592 20788 consensus_queue.cc:237] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:39.884563 20210 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.884625 18399 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.885203 21546 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.885663 20918 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.886694 18529 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.887459 18528 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:39.887537 21546 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.888188 21316 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.902832 20918 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.906293 18529 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.909945 18399 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.910669 20210 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.911048 18119 catalog_manager.cc:5484] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:39.912413 21420 raft_consensus.cc:1050] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 to VOTER
I20240503 14:14:39.913424 21420 consensus_queue.cc:237] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:39.916338 18399 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.916630 18657 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.917250 20209 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.917243 20845 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.917891 20845 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.921509 21042 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.924786 18660 consensus_queue.cc:237] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:39.928947 20209 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.929558 18528 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:39.929908 20918 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.930377 21546 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:39.933888 21546 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.941809 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9577e6d3801f42f89afe8a48a77031dd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:39.942576 20845 raft_consensus.cc:1050] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 to VOTER
I20240503 14:14:39.943449 20845 consensus_queue.cc:237] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:39.943956 20209 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.944996 18119 catalog_manager.cc:5484] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:39.949877 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet 9577e6d3801f42f89afe8a48a77031dd with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:39.950296 21480 tablet_replica.cc:330] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:39.950506 21480 raft_consensus.cc:2229] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:39.950767 21480 raft_consensus.cc:2258] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:39.952281 21480 ts_tablet_manager.cc:1875] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:39.963878 21042 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.964807 18399 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.966017 20209 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.966970 21480 ts_tablet_manager.cc:1888] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:39.967532 21480 log.cc:1196] T 9577e6d3801f42f89afe8a48a77031dd P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/9577e6d3801f42f89afe8a48a77031dd
I20240503 14:14:39.968902 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet 9577e6d3801f42f89afe8a48a77031dd (table rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]) successfully deleted
I20240503 14:14:39.970875 18529 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.972539 18119 catalog_manager.cc:5484] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 reported cstate change: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.972980 18657 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:39.982060 20342 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.982842 20717 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.988619 18399 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.989360 20845 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:39.992008 18528 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:39.992453 21420 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.006894 21420 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.012217 20342 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.013306 18120 catalog_manager.cc:5484] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.019346 18528 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.022418 21672 raft_consensus.cc:1050] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER a632094ba4554a4895d6c0b538b81af8 to VOTER
I20240503 14:14:40.023355 21672 consensus_queue.cc:237] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:40.028081 21478 raft_consensus.cc:1050] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722: attempting to promote NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 to VOTER
I20240503 14:14:40.028954 21478 consensus_queue.cc:237] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:40.044796 20475 raft_consensus.cc:1261] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.044806 18657 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.045519 21671 consensus_queue.cc:1035] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.046103 18267 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.046418 21485 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.047106 20342 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.048233 21485 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.048586 18398 raft_consensus.cc:1261] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.048771 21485 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.049927 21671 consensus_queue.cc:1035] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.050715 18267 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:40.051942 21316 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.060295 18400 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.063434 18528 raft_consensus.cc:1261] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.064007 21482 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.066040 20918 consensus_queue.cc:1035] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.066709 20788 raft_consensus.cc:2932] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:40.066931 18398 raft_consensus.cc:2932] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:40.068786 18660 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.073514 18657 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:40.074550 20342 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.076774 18121 catalog_manager.cc:5484] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:40.077788 21687 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.078032 18267 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.082196 18269 consensus_queue.cc:237] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:40.085439 18400 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.085883 20342 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.086894 20717 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.087576 20717 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.091858 18530 consensus_queue.cc:237] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:40.091814 20717 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.099443 20341 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.101073 20342 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.101653 18398 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.103730 18120 catalog_manager.cc:5484] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:40.105646 18657 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.105854 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:40.107998 18128 catalog_manager.cc:5484] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.101136 21485 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20240503 14:14:40.109640 20475 raft_consensus.cc:2932] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:40.110620 21414 raft_consensus.cc:1050] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 to VOTER
I20240503 14:14:40.111464 21414 consensus_queue.cc:237] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:40.114593 21485 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.115993 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:40.118697 21485 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.122180 21696 tablet_replica.cc:330] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:40.122412 21696 raft_consensus.cc:2229] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:40.123082 21696 raft_consensus.cc:2258] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:40.122813 20342 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.124788 21696 ts_tablet_manager.cc:1875] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:40.126308 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 19b1b6be3bda47829f53765282eae9f9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:40.127759 18128 catalog_manager.cc:5484] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:40.134946 20209 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.135879 18657 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:40.139308 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b1b6be3bda47829f53765282eae9f9 with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:40.139735 21554 tablet_replica.cc:330] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:40.139978 21554 raft_consensus.cc:2229] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:40.140017 21042 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.140316 21554 raft_consensus.cc:2258] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:40.140627 18398 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.141310 21042 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.141831 21554 ts_tablet_manager.cc:1875] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:40.134687 18529 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.148736 21420 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.154425 21554 ts_tablet_manager.cc:1888] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:40.154713 21554 log.cc:1196] T 19b1b6be3bda47829f53765282eae9f9 P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/19b1b6be3bda47829f53765282eae9f9
I20240503 14:14:40.155771 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet 19b1b6be3bda47829f53765282eae9f9 (table rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]) successfully deleted
I20240503 14:14:40.156813 21696 ts_tablet_manager.cc:1888] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:40.159966 21696 log.cc:1196] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/a39fbf1d8f6f46ccab3c2b992ed0c63e
I20240503 14:14:40.161532 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet a39fbf1d8f6f46ccab3c2b992ed0c63e (table rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]) successfully deleted
I20240503 14:14:40.170495 18528 raft_consensus.cc:2932] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:40.181582 21485 raft_consensus.cc:1050] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722: attempting to promote NON_VOTER 80260781ae364d049c2500ff76a06ce8 to VOTER
I20240503 14:14:40.182499 21485 consensus_queue.cc:237] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:40.185586 21042 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:40.186408 18398 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:40.187712 20604 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.188275 18657 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.189393 21482 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.190079 21482 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.191048 20209 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:40.192231 18128 catalog_manager.cc:5484] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:40.197300 18268 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.199259 21478 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.203119 18269 consensus_queue.cc:237] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:40.207302 20209 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.207880 21042 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.209535 18398 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.210379 21042 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.213359 21420 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:40.214066 18398 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:40.216377 21703 raft_consensus.cc:1050] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: attempting to promote NON_VOTER 80260781ae364d049c2500ff76a06ce8 to VOTER
I20240503 14:14:40.217355 21703 consensus_queue.cc:237] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:40.217736 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eedc3cfb2dff4f9084446d89b9b3c0f4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:40.220324 18120 catalog_manager.cc:5484] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.224028 20605 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.224288 18269 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.225266 20210 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:40.225816 18657 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:40.226516 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet eedc3cfb2dff4f9084446d89b9b3c0f4 with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:40.226882 21696 tablet_replica.cc:330] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:40.227087 21696 raft_consensus.cc:2229] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:40.227267 21696 pending_rounds.cc:62] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: Trying to abort 1 pending ops.
I20240503 14:14:40.227372 21696 pending_rounds.cc:68] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023598305734258688 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "eedc3cfb2dff4f9084446d89b9b3c0f4" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } } }
I20240503 14:14:40.227840 21696 raft_consensus.cc:2894] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 14:14:40.227954 21696 raft_consensus.cc:2866] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:14:40.228060 21696 raft_consensus.cc:2258] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:40.229159 21696 ts_tablet_manager.cc:1875] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:40.229584 20392 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.230226 20392 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.230769 20392 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:40.235770 21715 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.237649 21696 ts_tablet_manager.cc:1888] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:40.237872 21696 log.cc:1196] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/eedc3cfb2dff4f9084446d89b9b3c0f4
I20240503 14:14:40.238678 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet eedc3cfb2dff4f9084446d89b9b3c0f4 (table rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]) successfully deleted
I20240503 14:14:40.240134 20604 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.240725 18269 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.243115 18657 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.244556 21485 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.246390 18120 catalog_manager.cc:5484] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:40.247849 20605 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.253069 18398 consensus_queue.cc:237] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:40.256805 18269 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.254942 18128 catalog_manager.cc:5484] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 reported cstate change: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.257426 21697 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.258006 18268 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.260237 21715 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.259850 18657 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.261077 18269 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.265111 18114 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c337ef0405e4856bd2a608a6a602a49 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:40.265980 18529 consensus_queue.cc:237] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:40.267627 18119 catalog_manager.cc:5484] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.269196 20603 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.270689 21485 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 14:14:40.271560 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c337ef0405e4856bd2a608a6a602a49 with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:40.272528 21723 tablet_replica.cc:330] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:40.272840 21723 raft_consensus.cc:2229] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:40.273758 18269 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.274868 20605 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.275769 21715 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.277194 21482 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.277792 21482 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.278268 21723 raft_consensus.cc:2258] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:40.278035 20604 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.279556 21723 ts_tablet_manager.cc:1875] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:40.281821 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c35376539bc944e59b74bfe59d5feb9e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:40.282615 20605 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.285082 18268 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.286954 21723 ts_tablet_manager.cc:1888] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:40.286763 18120 catalog_manager.cc:5484] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:40.287391 21723 log.cc:1196] T 8c337ef0405e4856bd2a608a6a602a49 P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/8c337ef0405e4856bd2a608a6a602a49
I20240503 14:14:40.288254 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet 8c337ef0405e4856bd2a608a6a602a49 (table rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]) successfully deleted
I20240503 14:14:40.292315 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet c35376539bc944e59b74bfe59d5feb9e with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:40.292703 21723 tablet_replica.cc:330] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:40.292986 21723 raft_consensus.cc:2229] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:40.293396 21723 raft_consensus.cc:2258] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:40.294823 21723 ts_tablet_manager.cc:1875] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:40.301347 21723 ts_tablet_manager.cc:1888] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:40.301613 21723 log.cc:1196] T c35376539bc944e59b74bfe59d5feb9e P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/c35376539bc944e59b74bfe59d5feb9e
I20240503 14:14:40.302986 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet c35376539bc944e59b74bfe59d5feb9e (table rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]) successfully deleted
I20240503 14:14:40.320076 18268 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91743575cf4a4f38b5b43265539237a3"
dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58274
I20240503 14:14:40.320377 18268 raft_consensus.cc:595] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:40.342211 18398 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0c1a7f5739f4fa59271254e501fb095"
dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45784
I20240503 14:14:40.342499 18398 raft_consensus.cc:595] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:40.364398 21697 raft_consensus.cc:979] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: : Instructing follower 69cbbbedfe03437b8c95b7bb730dbe66 to start an election
I20240503 14:14:40.364639 21697 raft_consensus.cc:1067] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Signalling peer 69cbbbedfe03437b8c95b7bb730dbe66 to start an election
I20240503 14:14:40.365691 20342 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0c1a7f5739f4fa59271254e501fb095"
dest_uuid: "69cbbbedfe03437b8c95b7bb730dbe66"
from {username='slave'} at 127.10.145.2:36685
I20240503 14:14:40.365980 20342 raft_consensus.cc:482] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:40.366221 20342 raft_consensus.cc:3037] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.368738 20342 raft_consensus.cc:504] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:40.372822 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1a7f5739f4fa59271254e501fb095" candidate_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:40.373114 18398 raft_consensus.cc:3032] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:40.373044 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1a7f5739f4fa59271254e501fb095" candidate_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:40.373208 18398 raft_consensus.cc:729] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Running, Role: LEADER
I20240503 14:14:40.373315 18529 raft_consensus.cc:3037] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.373675 18398 consensus_queue.cc:260] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:40.374061 18398 raft_consensus.cc:3037] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.375528 18529 raft_consensus.cc:2454] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69cbbbedfe03437b8c95b7bb730dbe66 in term 2.
I20240503 14:14:40.376241 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1a7f5739f4fa59271254e501fb095" candidate_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:40.376498 18269 raft_consensus.cc:3037] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.377877 18398 raft_consensus.cc:2454] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69cbbbedfe03437b8c95b7bb730dbe66 in term 2.
I20240503 14:14:40.378437 20277 leader_election.cc:302] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 69cbbbedfe03437b8c95b7bb730dbe66, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:40.378908 21277 raft_consensus.cc:2781] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:40.379328 18269 raft_consensus.cc:2454] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69cbbbedfe03437b8c95b7bb730dbe66 in term 2.
I20240503 14:14:40.379835 20342 leader_election.cc:288] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [CANDIDATE]: Term 2 election: Requested vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:40.380909 21277 raft_consensus.cc:686] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 LEADER]: Becoming Leader. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Running, Role: LEADER
I20240503 14:14:40.381457 21277 consensus_queue.cc:237] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:40.385704 18119 catalog_manager.cc:5484] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: term changed from 1 to 2, leader changed from ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) to 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6). New cstate: current_term: 2 leader_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:40.399019 21763 ts_tablet_manager.cc:902] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:40.400794 21763 tablet_copy_client.cc:308] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:40.401746 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet 10a4ca83469d4dab823a26d6cbe952a6 from peer 69cbbbedfe03437b8c95b7bb730dbe66 ({username='slave'} at 127.10.145.6:41609)
I20240503 14:14:40.401966 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet 10a4ca83469d4dab823a26d6cbe952a6 from peer 69cbbbedfe03437b8c95b7bb730dbe66 at {username='slave'} at 127.10.145.6:41609: session id = 69cbbbedfe03437b8c95b7bb730dbe66-10a4ca83469d4dab823a26d6cbe952a6
I20240503 14:14:40.405103 18289 tablet_copy_source_session.cc:205] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:40.407023 21763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a4ca83469d4dab823a26d6cbe952a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:40.413313 21763 tablet_copy_client.cc:783] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:40.413587 21763 tablet_copy_client.cc:647] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:40.415490 21768 tablet_copy_client.cc:1002] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:40.443358 20845 raft_consensus.cc:979] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: : Instructing follower 80260781ae364d049c2500ff76a06ce8 to start an election
I20240503 14:14:40.443544 20845 raft_consensus.cc:1067] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Signalling peer 80260781ae364d049c2500ff76a06ce8 to start an election
I20240503 14:14:40.444264 20605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91743575cf4a4f38b5b43265539237a3"
dest_uuid: "80260781ae364d049c2500ff76a06ce8"
from {username='slave'} at 127.10.145.1:56471
I20240503 14:14:40.444520 20605 raft_consensus.cc:482] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:40.444682 20605 raft_consensus.cc:3037] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.447245 20605 raft_consensus.cc:504] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:40.448881 20605 leader_election.cc:288] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [CANDIDATE]: Term 2 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:40.449055 18657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91743575cf4a4f38b5b43265539237a3" candidate_uuid: "80260781ae364d049c2500ff76a06ce8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:40.449424 18657 raft_consensus.cc:3037] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.449493 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91743575cf4a4f38b5b43265539237a3" candidate_uuid: "80260781ae364d049c2500ff76a06ce8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:40.449702 18269 raft_consensus.cc:3032] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:40.449818 18269 raft_consensus.cc:729] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:40.450112 18269 consensus_queue.cc:260] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:40.450608 18269 raft_consensus.cc:3037] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.452436 18657 raft_consensus.cc:2454] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80260781ae364d049c2500ff76a06ce8 in term 2.
I20240503 14:14:40.453544 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91743575cf4a4f38b5b43265539237a3" candidate_uuid: "80260781ae364d049c2500ff76a06ce8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:40.453758 18398 raft_consensus.cc:3037] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.455907 18398 raft_consensus.cc:2454] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80260781ae364d049c2500ff76a06ce8 in term 2.
I20240503 14:14:40.456279 18269 raft_consensus.cc:2454] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80260781ae364d049c2500ff76a06ce8 in term 2.
I20240503 14:14:40.456435 20540 leader_election.cc:302] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 80260781ae364d049c2500ff76a06ce8, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:40.456812 21690 raft_consensus.cc:2781] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:40.456961 21690 raft_consensus.cc:686] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 2 LEADER]: Becoming Leader. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Running, Role: LEADER
I20240503 14:14:40.457396 21690 consensus_queue.cc:237] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:40.461454 18119 catalog_manager.cc:5484] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 reported cstate change: term changed from 1 to 2, leader changed from 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) to 80260781ae364d049c2500ff76a06ce8 (127.10.145.8). New cstate: current_term: 2 leader_uuid: "80260781ae364d049c2500ff76a06ce8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:40.551681 18269 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "79373bd79a934eaab67bbcf2208e7b5b"
dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58290
I20240503 14:14:40.551955 18269 raft_consensus.cc:595] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:40.565109 18398 consensus_queue.cc:237] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:40.567746 18657 raft_consensus.cc:1261] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.567862 18529 raft_consensus.cc:1261] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:40.567885 18336 consensus_peers.cc:485] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 -> Peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Couldn't send request to peer 2e6a3f22c7ae444e96f6d35cca602e62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5b74777c90f4cb7937d0dfc54420467. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.568447 21715 consensus_queue.cc:1035] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.568753 21697 consensus_queue.cc:1035] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.571369 21702 raft_consensus.cc:2932] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:40.572054 18529 raft_consensus.cc:2932] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:40.575666 18657 raft_consensus.cc:2932] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:40.577186 18119 catalog_manager.cc:5484] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:40.591749 18657 consensus_queue.cc:237] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
W20240503 14:14:40.594398 18596 consensus_peers.cc:485] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 -> Peer a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307): Couldn't send request to peer a632094ba4554a4895d6c0b538b81af8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2167226099a4dcf98304918e8f412af. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.594357 18398 raft_consensus.cc:1261] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.594590 18269 raft_consensus.cc:1261] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.595073 21671 consensus_queue.cc:1035] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.595388 21672 consensus_queue.cc:1035] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.598353 20918 raft_consensus.cc:2932] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:40.599051 18398 raft_consensus.cc:2932] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:40.602351 18269 raft_consensus.cc:2932] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:40.604866 18128 catalog_manager.cc:5484] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index -1 to 4, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:40.624023 18657 consensus_queue.cc:237] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:40.626850 18398 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.627267 18269 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.627995 21671 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.628451 20788 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.632056 21671 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:40.633808 18398 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:40.635916 18269 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:40.640172 18128 catalog_manager.cc:5484] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index -1 to 4, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
W20240503 14:14:40.644232 18594 consensus_peers.cc:485] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 -> Peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Couldn't send request to peer 69cbbbedfe03437b8c95b7bb730dbe66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bc67ebb58a246d9a8c47f0838a4831a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.651892 18657 consensus_queue.cc:237] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:40.654032 18529 raft_consensus.cc:1261] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.654101 20475 raft_consensus.cc:1261] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:40.654667 20918 consensus_queue.cc:1035] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.655326 20918 consensus_queue.cc:1035] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.657864 21672 raft_consensus.cc:2932] T 4d24e31ff3ad41d98214bb41d8c28fda P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:40.658557 20475 raft_consensus.cc:2932] T 4d24e31ff3ad41d98214bb41d8c28fda P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:40.658537 18529 raft_consensus.cc:2932] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:40.661382 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4d24e31ff3ad41d98214bb41d8c28fda with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:40.662879 18269 consensus_queue.cc:237] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:40.664057 18120 catalog_manager.cc:5484] T 4d24e31ff3ad41d98214bb41d8c28fda P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:40.666488 18657 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.667038 20845 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:40.667655 18202 consensus_peers.cc:485] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 -> Peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107): Couldn't send request to peer 80260781ae364d049c2500ff76a06ce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cb29bdb71f24b46b32e7b97d0172c20. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.667943 18528 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.668989 20845 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.671376 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d24e31ff3ad41d98214bb41d8c28fda with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:40.671869 20845 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:40.672394 18657 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:40.680014 18128 catalog_manager.cc:5484] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.683703 21856 tablet_replica.cc:330] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:40.684053 21856 raft_consensus.cc:2229] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:40.684532 21856 raft_consensus.cc:2258] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:40.685808 21856 ts_tablet_manager.cc:1875] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:40.694023 21856 ts_tablet_manager.cc:1888] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:40.694247 21856 log.cc:1196] T 4d24e31ff3ad41d98214bb41d8c28fda P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/4d24e31ff3ad41d98214bb41d8c28fda
I20240503 14:14:40.695289 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet 4d24e31ff3ad41d98214bb41d8c28fda (table rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]) successfully deleted
I20240503 14:14:40.696151 18528 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:40.705057 20605 consensus_queue.cc:237] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:40.707185 18657 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:40.707278 18528 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:40.707746 21690 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:40.708115 21687 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:40.710639 21690 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.711117 18528 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.714864 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea3ea2e3d87f4c099abbf5193c239a8b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:40.716725 18657 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.717483 18120 catalog_manager.cc:5484] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 2 leader_uuid: "80260781ae364d049c2500ff76a06ce8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:40.721266 18269 consensus_queue.cc:237] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:40.723569 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3ea2e3d87f4c099abbf5193c239a8b with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:40.726119 18528 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:40.726251 18205 consensus_peers.cc:485] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 -> Peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Couldn't send request to peer 2e6a3f22c7ae444e96f6d35cca602e62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe3d20ad69474bcbae19bdd06cbfeb52. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.726874 18398 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.727046 21420 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.727558 20845 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.731030 21414 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:40.736855 21866 tablet_replica.cc:330] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:40.735795 18528 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:40.737264 21866 raft_consensus.cc:2229] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:40.735090 18398 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:40.737751 21866 raft_consensus.cc:2258] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:40.738783 21866 ts_tablet_manager.cc:1875] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:40.745210 21866 ts_tablet_manager.cc:1888] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:14:40.745462 21866 log.cc:1196] T ea3ea2e3d87f4c099abbf5193c239a8b P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/ea3ea2e3d87f4c099abbf5193c239a8b
I20240503 14:14:40.745501 18119 catalog_manager.cc:5484] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.746321 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet ea3ea2e3d87f4c099abbf5193c239a8b (table rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]) successfully deleted
I20240503 14:14:40.762709 18657 consensus_queue.cc:237] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:40.764954 18528 raft_consensus.cc:1261] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.765045 18398 raft_consensus.cc:1261] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:40.765383 18593 consensus_peers.cc:485] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 -> Peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107): Couldn't send request to peer 80260781ae364d049c2500ff76a06ce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f09411ea6c414451b66fd457d9aaf010. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.765599 20788 consensus_queue.cc:1035] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.765898 21671 consensus_queue.cc:1035] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.768831 20918 raft_consensus.cc:2932] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:40.769569 18398 raft_consensus.cc:2932] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:40.769820 18528 raft_consensus.cc:2932] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:40.774722 18119 catalog_manager.cc:5484] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.786972 18269 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:40.787622 21277 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.788216 18269 consensus_queue.cc:237] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:40.798879 18398 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:40.799091 18528 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:40.799499 18205 consensus_peers.cc:485] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 -> Peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Couldn't send request to peer 2e6a3f22c7ae444e96f6d35cca602e62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2c5c6f4317649fab75d01e339b76403. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:40.799633 21042 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.799957 20845 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:40.802771 21042 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:40.803357 18528 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:40.810050 18120 catalog_manager.cc:5484] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.819000 18400 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:40.860674 21635 heartbeater.cc:503] Master 127.10.145.62:33209 was elected leader, sending a full tablet report...
I20240503 14:14:40.867652 18657 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:40.868227 21863 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.871253 21042 raft_consensus.cc:979] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: : Instructing follower 1c72210d694344d4b4604140088a6471 to start an election
I20240503 14:14:40.871412 21042 raft_consensus.cc:1067] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Signalling peer 1c72210d694344d4b4604140088a6471 to start an election
I20240503 14:14:40.872393 18657 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "79373bd79a934eaab67bbcf2208e7b5b"
dest_uuid: "1c72210d694344d4b4604140088a6471"
from {username='slave'} at 127.10.145.1:50073
I20240503 14:14:40.872638 18657 raft_consensus.cc:482] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:40.872716 18657 raft_consensus.cc:3037] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.875443 18657 raft_consensus.cc:504] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:40.876674 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79373bd79a934eaab67bbcf2208e7b5b" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:40.876766 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79373bd79a934eaab67bbcf2208e7b5b" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:40.876896 18398 raft_consensus.cc:3037] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.876952 18269 raft_consensus.cc:3032] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:40.877022 18269 raft_consensus.cc:729] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:40.877257 18269 consensus_queue.cc:260] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:40.877689 18269 raft_consensus.cc:3037] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.880059 18269 raft_consensus.cc:2454] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 2.
I20240503 14:14:40.880847 18398 raft_consensus.cc:2454] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 2.
I20240503 14:14:40.880836 20210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79373bd79a934eaab67bbcf2208e7b5b" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2e6a3f22c7ae444e96f6d35cca602e62"
I20240503 14:14:40.881295 20210 raft_consensus.cc:3037] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.881353 18657 leader_election.cc:288] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 2 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605)
I20240503 14:14:40.881481 18595 leader_election.cc:302] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:40.881807 21671 raft_consensus.cc:2781] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:40.881961 21671 raft_consensus.cc:686] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 2 LEADER]: Becoming Leader. State: Replica: 1c72210d694344d4b4604140088a6471, State: Running, Role: LEADER
I20240503 14:14:40.882256 21671 consensus_queue.cc:237] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:40.884122 20210 raft_consensus.cc:2454] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 2.
I20240503 14:14:40.886106 18128 catalog_manager.cc:5484] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 reported cstate change: term changed from 1 to 2, leader changed from 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) to 1c72210d694344d4b4604140088a6471 (127.10.145.4). New cstate: current_term: 2 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:40.929919 18398 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:40.930646 21277 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:40.945014 18528 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:40.945791 21277 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.005678 18269 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:41.006448 21863 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.022586 21962 ts_tablet_manager.cc:902] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:41.024528 18398 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:41.025373 21863 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.062958 20605 consensus_queue.cc:237] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:41.066509 18400 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:41.067314 21863 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:41.068797 18660 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:41.070214 21863 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:41.072314 21863 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:41.072777 18400 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:41.076689 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 91743575cf4a4f38b5b43265539237a3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:41.077596 18660 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:41.078923 18119 catalog_manager.cc:5484] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 2 leader_uuid: "80260781ae364d049c2500ff76a06ce8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:41.083403 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet 91743575cf4a4f38b5b43265539237a3 with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:41.083956 21866 tablet_replica.cc:330] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:41.084164 21866 raft_consensus.cc:2229] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:41.084400 21866 raft_consensus.cc:2258] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:41.089071 21866 ts_tablet_manager.cc:1875] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:41.091670 21962 tablet_copy_client.cc:308] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:41.092454 18680 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet c2167226099a4dcf98304918e8f412af from peer a632094ba4554a4895d6c0b538b81af8 ({username='slave'} at 127.10.145.7:47505)
I20240503 14:14:41.092657 18680 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet c2167226099a4dcf98304918e8f412af from peer a632094ba4554a4895d6c0b538b81af8 at {username='slave'} at 127.10.145.7:47505: session id = a632094ba4554a4895d6c0b538b81af8-c2167226099a4dcf98304918e8f412af
I20240503 14:14:41.095506 18680 tablet_copy_source_session.cc:205] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.096452 21866 ts_tablet_manager.cc:1888] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:14:41.096601 21866 log.cc:1196] T 91743575cf4a4f38b5b43265539237a3 P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/91743575cf4a4f38b5b43265539237a3
I20240503 14:14:41.096909 21962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2167226099a4dcf98304918e8f412af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.097549 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet 91743575cf4a4f38b5b43265539237a3 (table rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]) successfully deleted
I20240503 14:14:41.102471 21962 tablet_copy_client.cc:783] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.102726 21962 tablet_copy_client.cc:647] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.104532 21980 tablet_copy_client.cc:1002] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:41.111441 18660 consensus_queue.cc:237] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:41.113847 18528 raft_consensus.cc:1261] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:41.114070 18400 raft_consensus.cc:1261] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:41.114106 18593 consensus_peers.cc:485] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 -> Peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107): Couldn't send request to peer 80260781ae364d049c2500ff76a06ce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3de72df2a82e445f9aa783bc741fa996. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:41.114487 21671 consensus_queue.cc:1035] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:41.114832 20918 consensus_queue.cc:1035] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:41.117461 21671 raft_consensus.cc:2932] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:41.118163 18400 raft_consensus.cc:2932] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:41.118317 18528 raft_consensus.cc:2932] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:41.122999 18119 catalog_manager.cc:5484] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index -1 to 4, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:41.130627 21994 ts_tablet_manager.cc:902] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: Initiating tablet copy from peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:41.133066 21994 tablet_copy_client.cc:308] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:43191
I20240503 14:14:41.133821 18420 tablet_copy_service.cc:140] P ea366e6e15a24c52b1ea5a154f942568: Received BeginTabletCopySession request for tablet d5b74777c90f4cb7937d0dfc54420467 from peer 2e6a3f22c7ae444e96f6d35cca602e62 ({username='slave'} at 127.10.145.5:47647)
I20240503 14:14:41.134438 18420 tablet_copy_service.cc:161] P ea366e6e15a24c52b1ea5a154f942568: Beginning new tablet copy session on tablet d5b74777c90f4cb7937d0dfc54420467 from peer 2e6a3f22c7ae444e96f6d35cca602e62 at {username='slave'} at 127.10.145.5:47647: session id = 2e6a3f22c7ae444e96f6d35cca602e62-d5b74777c90f4cb7937d0dfc54420467
I20240503 14:14:41.136950 18420 tablet_copy_source_session.cc:205] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.138309 21994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b74777c90f4cb7937d0dfc54420467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.139240 18528 consensus_queue.cc:237] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:41.142177 18400 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:41.142166 18660 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:41.142829 21482 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:41.143441 21482 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:41.144363 21994 tablet_copy_client.cc:783] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.144629 21994 tablet_copy_client.cc:647] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.146881 21998 tablet_copy_client.cc:1002] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:41.146762 21482 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:41.147439 18400 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
W20240503 14:14:41.153717 18466 consensus_peers.cc:485] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 -> Peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Couldn't send request to peer 2e6a3f22c7ae444e96f6d35cca602e62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71a326db29114366951a67b9aa84ac03. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:41.153649 18128 catalog_manager.cc:5484] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:41.164824 18660 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:41.177871 22009 ts_tablet_manager.cc:902] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:41.180377 22009 tablet_copy_client.cc:308] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:41.181365 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet 4cb29bdb71f24b46b32e7b97d0172c20 from peer 80260781ae364d049c2500ff76a06ce8 ({username='slave'} at 127.10.145.8:59415)
I20240503 14:14:41.181537 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet 4cb29bdb71f24b46b32e7b97d0172c20 from peer 80260781ae364d049c2500ff76a06ce8 at {username='slave'} at 127.10.145.8:59415: session id = 80260781ae364d049c2500ff76a06ce8-4cb29bdb71f24b46b32e7b97d0172c20
I20240503 14:14:41.184140 18289 tablet_copy_source_session.cc:205] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.185698 22009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb29bdb71f24b46b32e7b97d0172c20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.190517 22009 tablet_copy_client.cc:783] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.190874 22009 tablet_copy_client.cc:647] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.193420 22013 tablet_copy_client.cc:1002] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:41.229717 22034 ts_tablet_manager.cc:902] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:41.231364 22034 tablet_copy_client.cc:308] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:41.242749 22046 ts_tablet_manager.cc:902] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:41.249169 22046 tablet_copy_client.cc:308] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:41.249981 18680 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet f09411ea6c414451b66fd457d9aaf010 from peer 80260781ae364d049c2500ff76a06ce8 ({username='slave'} at 127.10.145.8:39351)
I20240503 14:14:41.250169 18680 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet f09411ea6c414451b66fd457d9aaf010 from peer 80260781ae364d049c2500ff76a06ce8 at {username='slave'} at 127.10.145.8:39351: session id = 80260781ae364d049c2500ff76a06ce8-f09411ea6c414451b66fd457d9aaf010
I20240503 14:14:41.250393 18679 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet 5bc67ebb58a246d9a8c47f0838a4831a from peer 69cbbbedfe03437b8c95b7bb730dbe66 ({username='slave'} at 127.10.145.6:36919)
I20240503 14:14:41.250566 18679 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet 5bc67ebb58a246d9a8c47f0838a4831a from peer 69cbbbedfe03437b8c95b7bb730dbe66 at {username='slave'} at 127.10.145.6:36919: session id = 69cbbbedfe03437b8c95b7bb730dbe66-5bc67ebb58a246d9a8c47f0838a4831a
I20240503 14:14:41.253573 18680 tablet_copy_source_session.cc:205] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.253654 18679 tablet_copy_source_session.cc:205] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.255090 22046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09411ea6c414451b66fd457d9aaf010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.255421 22034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc67ebb58a246d9a8c47f0838a4831a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.261502 22046 tablet_copy_client.cc:783] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.261633 22034 tablet_copy_client.cc:783] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.261799 22046 tablet_copy_client.cc:647] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.261951 22034 tablet_copy_client.cc:647] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.263870 22054 tablet_copy_client.cc:1002] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:41.264411 22040 tablet_copy_client.cc:1002] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:41.265623 22056 ts_tablet_manager.cc:902] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:41.267822 22056 tablet_copy_client.cc:308] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:41.268548 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet fe3d20ad69474bcbae19bdd06cbfeb52 from peer 2e6a3f22c7ae444e96f6d35cca602e62 ({username='slave'} at 127.10.145.5:45257)
I20240503 14:14:41.268724 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet fe3d20ad69474bcbae19bdd06cbfeb52 from peer 2e6a3f22c7ae444e96f6d35cca602e62 at {username='slave'} at 127.10.145.5:45257: session id = 2e6a3f22c7ae444e96f6d35cca602e62-fe3d20ad69474bcbae19bdd06cbfeb52
I20240503 14:14:41.270983 18289 tablet_copy_source_session.cc:205] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.272389 22056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3d20ad69474bcbae19bdd06cbfeb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.278934 22056 tablet_copy_client.cc:783] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.279243 22056 tablet_copy_client.cc:647] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.281086 22064 tablet_copy_client.cc:1002] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:41.299172 18400 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:41.299746 21546 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.324863 20210 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:41.325664 21546 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.331360 18269 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:14:41.335309 20918 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:41.337945 18660 consensus_queue.cc:237] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:41.340865 18400 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:41.341645 21671 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:41.344600 20209 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:41.346351 20918 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:41.349233 20788 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:41.350652 18400 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:41.353627 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 79373bd79a934eaab67bbcf2208e7b5b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:41.355031 20209 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:41.355686 18128 catalog_manager.cc:5484] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 5 to 7, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 2 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:41.359916 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet 79373bd79a934eaab67bbcf2208e7b5b with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:41.361639 21866 tablet_replica.cc:330] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:41.361863 21866 raft_consensus.cc:2229] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:41.362074 21866 raft_consensus.cc:2258] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:41.362962 21866 ts_tablet_manager.cc:1875] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:41.368741 21866 ts_tablet_manager.cc:1888] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:14:41.368870 21866 log.cc:1196] T 79373bd79a934eaab67bbcf2208e7b5b P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/79373bd79a934eaab67bbcf2208e7b5b
I20240503 14:14:41.371222 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet 79373bd79a934eaab67bbcf2208e7b5b (table rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]) successfully deleted
I20240503 14:14:41.376338 20342 consensus_queue.cc:237] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:41.378477 18269 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:41.378500 18529 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:41.379091 21277 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:41.379895 21935 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:41.381986 21277 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:41.382648 18529 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:41.385610 18269 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:41.387976 18113 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f0c1a7f5739f4fa59271254e501fb095 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:41.388865 18128 catalog_manager.cc:5484] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 5 to 7, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 2 leader_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:41.393754 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c1a7f5739f4fa59271254e501fb095 with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:41.398075 22111 ts_tablet_manager.cc:902] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:41.406929 22108 tablet_replica.cc:330] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:41.407375 22108 raft_consensus.cc:2229] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:41.407745 22108 raft_consensus.cc:2258] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:41.409101 22108 ts_tablet_manager.cc:1875] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:41.417405 22108 ts_tablet_manager.cc:1888] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:14:41.418064 22108 log.cc:1196] T f0c1a7f5739f4fa59271254e501fb095 P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/f0c1a7f5739f4fa59271254e501fb095
I20240503 14:14:41.418867 22111 tablet_copy_client.cc:308] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:41.419752 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet f2c5c6f4317649fab75d01e339b76403 from peer 2e6a3f22c7ae444e96f6d35cca602e62 ({username='slave'} at 127.10.145.5:45257)
I20240503 14:14:41.419883 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet f0c1a7f5739f4fa59271254e501fb095 (table rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]) successfully deleted
I20240503 14:14:41.419932 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet f2c5c6f4317649fab75d01e339b76403 from peer 2e6a3f22c7ae444e96f6d35cca602e62 at {username='slave'} at 127.10.145.5:45257: session id = 2e6a3f22c7ae444e96f6d35cca602e62-f2c5c6f4317649fab75d01e339b76403
I20240503 14:14:41.422315 18289 tablet_copy_source_session.cc:205] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.423954 22111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c5c6f4317649fab75d01e339b76403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.429391 22111 tablet_copy_client.cc:783] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.429731 22111 tablet_copy_client.cc:647] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.431792 22122 tablet_copy_client.cc:1002] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:41.665735 22215 ts_tablet_manager.cc:902] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:41.667192 22215 tablet_copy_client.cc:308] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:41.668015 18679 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet 3de72df2a82e445f9aa783bc741fa996 from peer 80260781ae364d049c2500ff76a06ce8 ({username='slave'} at 127.10.145.8:39351)
I20240503 14:14:41.668198 18679 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet 3de72df2a82e445f9aa783bc741fa996 from peer 80260781ae364d049c2500ff76a06ce8 at {username='slave'} at 127.10.145.8:39351: session id = 80260781ae364d049c2500ff76a06ce8-3de72df2a82e445f9aa783bc741fa996
I20240503 14:14:41.670795 18679 tablet_copy_source_session.cc:205] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.672076 22215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3de72df2a82e445f9aa783bc741fa996. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.676636 22215 tablet_copy_client.cc:783] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.676934 22215 tablet_copy_client.cc:647] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.678628 22220 tablet_copy_client.cc:1002] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:41.753885 22221 ts_tablet_manager.cc:902] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: Initiating tablet copy from peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:41.755196 22221 tablet_copy_client.cc:308] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:38857
I20240503 14:14:41.761411 18550 tablet_copy_service.cc:140] P 55339e38e7aa4bb7b7b42aa206108722: Received BeginTabletCopySession request for tablet 71a326db29114366951a67b9aa84ac03 from peer 2e6a3f22c7ae444e96f6d35cca602e62 ({username='slave'} at 127.10.145.5:59583)
I20240503 14:14:41.761587 18550 tablet_copy_service.cc:161] P 55339e38e7aa4bb7b7b42aa206108722: Beginning new tablet copy session on tablet 71a326db29114366951a67b9aa84ac03 from peer 2e6a3f22c7ae444e96f6d35cca602e62 at {username='slave'} at 127.10.145.5:59583: session id = 2e6a3f22c7ae444e96f6d35cca602e62-71a326db29114366951a67b9aa84ac03
I20240503 14:14:41.763605 18550 tablet_copy_source_session.cc:205] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:41.764971 22221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a326db29114366951a67b9aa84ac03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:41.769515 22221 tablet_copy_client.cc:783] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:41.769786 22221 tablet_copy_client.cc:647] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:41.771481 22225 tablet_copy_client.cc:1002] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:41.915930 21763 tablet_copy_client.cc:523] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:41.921886 21763 tablet_bootstrap.cc:492] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap starting.
I20240503 14:14:41.960103 21763 tablet_bootstrap.cc:492] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:41.960523 21763 tablet_bootstrap.cc:492] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap complete.
I20240503 14:14:41.960793 21763 ts_tablet_manager.cc:1367] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent bootstrapping tablet: real 0.039s user 0.033s sys 0.001s
I20240503 14:14:41.961546 21763 raft_consensus.cc:348] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:41.961782 21763 raft_consensus.cc:729] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Initialized, Role: LEARNER
I20240503 14:14:41.961979 21763 consensus_queue.cc:260] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:41.962539 21763 ts_tablet_manager.cc:1398] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:41.963344 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-10a4ca83469d4dab823a26d6cbe952a6 received from {username='slave'} at 127.10.145.6:41609
I20240503 14:14:41.963492 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-10a4ca83469d4dab823a26d6cbe952a6 on tablet 10a4ca83469d4dab823a26d6cbe952a6 with peer 69cbbbedfe03437b8c95b7bb730dbe66
I20240503 14:14:42.388300 20342 raft_consensus.cc:1203] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:42.605068 21962 tablet_copy_client.cc:523] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:42.608764 21962 tablet_bootstrap.cc:492] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: Bootstrap starting.
I20240503 14:14:42.638729 21962 tablet_bootstrap.cc:492] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:42.639235 21962 tablet_bootstrap.cc:492] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: Bootstrap complete.
I20240503 14:14:42.639550 21962 ts_tablet_manager.cc:1367] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: Time spent bootstrapping tablet: real 0.031s user 0.032s sys 0.000s
I20240503 14:14:42.640314 21962 raft_consensus.cc:348] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:42.640520 21962 raft_consensus.cc:729] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a632094ba4554a4895d6c0b538b81af8, State: Initialized, Role: LEARNER
I20240503 14:14:42.640688 21962 consensus_queue.cc:260] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:42.641402 21962 ts_tablet_manager.cc:1398] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240503 14:14:42.642272 18679 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session a632094ba4554a4895d6c0b538b81af8-c2167226099a4dcf98304918e8f412af received from {username='slave'} at 127.10.145.7:47505
I20240503 14:14:42.642418 18679 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session a632094ba4554a4895d6c0b538b81af8-c2167226099a4dcf98304918e8f412af on tablet c2167226099a4dcf98304918e8f412af with peer a632094ba4554a4895d6c0b538b81af8
I20240503 14:14:42.647353 21994 tablet_copy_client.cc:523] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:42.650427 21994 tablet_bootstrap.cc:492] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap starting.
I20240503 14:14:42.684499 21994 tablet_bootstrap.cc:492] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:42.684916 21994 tablet_bootstrap.cc:492] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap complete.
I20240503 14:14:42.685173 21994 ts_tablet_manager.cc:1367] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.005s
I20240503 14:14:42.686026 21994 raft_consensus.cc:348] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:42.686269 21994 raft_consensus.cc:729] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e6a3f22c7ae444e96f6d35cca602e62, State: Initialized, Role: LEARNER
I20240503 14:14:42.686450 21994 consensus_queue.cc:260] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:42.687165 21994 ts_tablet_manager.cc:1398] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
I20240503 14:14:42.688127 18420 tablet_copy_service.cc:342] P ea366e6e15a24c52b1ea5a154f942568: Request end of tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-d5b74777c90f4cb7937d0dfc54420467 received from {username='slave'} at 127.10.145.5:47647
I20240503 14:14:42.688274 18420 tablet_copy_service.cc:433] P ea366e6e15a24c52b1ea5a154f942568: ending tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-d5b74777c90f4cb7937d0dfc54420467 on tablet d5b74777c90f4cb7937d0dfc54420467 with peer 2e6a3f22c7ae444e96f6d35cca602e62
I20240503 14:14:42.693868 22009 tablet_copy_client.cc:523] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:42.696638 22009 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: Bootstrap starting.
I20240503 14:14:42.729043 22009 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:42.729485 22009 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: Bootstrap complete.
I20240503 14:14:42.729748 22009 ts_tablet_manager.cc:1367] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.004s
I20240503 14:14:42.730489 22009 raft_consensus.cc:348] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:42.730759 22009 raft_consensus.cc:729] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Initialized, Role: LEARNER
I20240503 14:14:42.730954 22009 consensus_queue.cc:260] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:42.731495 22009 ts_tablet_manager.cc:1398] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:42.732350 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 80260781ae364d049c2500ff76a06ce8-4cb29bdb71f24b46b32e7b97d0172c20 received from {username='slave'} at 127.10.145.8:59415
I20240503 14:14:42.732519 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 80260781ae364d049c2500ff76a06ce8-4cb29bdb71f24b46b32e7b97d0172c20 on tablet 4cb29bdb71f24b46b32e7b97d0172c20 with peer 80260781ae364d049c2500ff76a06ce8
I20240503 14:14:42.764362 22046 tablet_copy_client.cc:523] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:42.764861 22034 tablet_copy_client.cc:523] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:42.767283 22046 tablet_bootstrap.cc:492] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: Bootstrap starting.
I20240503 14:14:42.767473 22034 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap starting.
I20240503 14:14:42.782145 22056 tablet_copy_client.cc:523] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:42.785516 22056 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap starting.
I20240503 14:14:42.795938 21414 raft_consensus.cc:1050] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 to VOTER
I20240503 14:14:42.796912 21414 consensus_queue.cc:237] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:42.812637 20342 raft_consensus.cc:1261] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.813588 18529 raft_consensus.cc:1261] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.815816 20475 raft_consensus.cc:1203] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:42.815800 21420 consensus_queue.cc:1035] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.816459 21420 consensus_queue.cc:1035] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.817516 18660 raft_consensus.cc:1261] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.818703 21042 consensus_queue.cc:1035] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.824900 22034 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:42.825846 22034 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap complete.
I20240503 14:14:42.826164 22034 ts_tablet_manager.cc:1367] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent bootstrapping tablet: real 0.059s user 0.046s sys 0.004s
I20240503 14:14:42.832376 22034 raft_consensus.cc:348] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:42.832690 22034 raft_consensus.cc:729] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Initialized, Role: LEARNER
I20240503 14:14:42.832963 22034 consensus_queue.cc:260] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:42.833786 22034 ts_tablet_manager.cc:1398] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:14:42.834066 22046 tablet_bootstrap.cc:492] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:42.834677 22046 tablet_bootstrap.cc:492] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: Bootstrap complete.
I20240503 14:14:42.834764 18679 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-5bc67ebb58a246d9a8c47f0838a4831a received from {username='slave'} at 127.10.145.6:36919
I20240503 14:14:42.834900 18679 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-5bc67ebb58a246d9a8c47f0838a4831a on tablet 5bc67ebb58a246d9a8c47f0838a4831a with peer 69cbbbedfe03437b8c95b7bb730dbe66
I20240503 14:14:42.835005 22046 ts_tablet_manager.cc:1367] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: Time spent bootstrapping tablet: real 0.068s user 0.046s sys 0.004s
I20240503 14:14:42.835819 22046 raft_consensus.cc:348] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:42.836073 22046 raft_consensus.cc:729] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Initialized, Role: LEARNER
I20240503 14:14:42.836320 22046 consensus_queue.cc:260] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:42.836941 22046 ts_tablet_manager.cc:1398] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:42.837780 18679 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session 80260781ae364d049c2500ff76a06ce8-f09411ea6c414451b66fd457d9aaf010 received from {username='slave'} at 127.10.145.8:39351
I20240503 14:14:42.837913 18679 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session 80260781ae364d049c2500ff76a06ce8-f09411ea6c414451b66fd457d9aaf010 on tablet f09411ea6c414451b66fd457d9aaf010 with peer 80260781ae364d049c2500ff76a06ce8
I20240503 14:14:42.841732 21042 raft_consensus.cc:2932] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:42.845958 18529 raft_consensus.cc:2932] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:42.846055 20341 raft_consensus.cc:2932] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:42.847553 18660 raft_consensus.cc:2932] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:42.852346 18120 catalog_manager.cc:5484] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:42.855782 20605 raft_consensus.cc:1203] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:42.866817 18269 consensus_queue.cc:237] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:42.869233 20341 raft_consensus.cc:1261] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:42.870133 18529 raft_consensus.cc:1261] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:42.870467 21414 consensus_queue.cc:1035] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:42.871416 21414 consensus_queue.cc:1035] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:42.874404 21042 raft_consensus.cc:2932] T 10a4ca83469d4dab823a26d6cbe952a6 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:42.875212 22056 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:42.875702 22056 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap complete.
I20240503 14:14:42.876040 22056 ts_tablet_manager.cc:1367] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent bootstrapping tablet: real 0.091s user 0.047s sys 0.000s
I20240503 14:14:42.875800 20341 raft_consensus.cc:2932] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:42.876909 22056 raft_consensus.cc:348] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:42.877169 22056 raft_consensus.cc:729] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e6a3f22c7ae444e96f6d35cca602e62, State: Initialized, Role: LEARNER
I20240503 14:14:42.877432 22056 consensus_queue.cc:260] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:42.878177 22056 ts_tablet_manager.cc:1398] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:42.878252 18529 raft_consensus.cc:2932] T 10a4ca83469d4dab823a26d6cbe952a6 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:42.878927 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-fe3d20ad69474bcbae19bdd06cbfeb52 received from {username='slave'} at 127.10.145.5:45257
I20240503 14:14:42.879088 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-fe3d20ad69474bcbae19bdd06cbfeb52 on tablet fe3d20ad69474bcbae19bdd06cbfeb52 with peer 2e6a3f22c7ae444e96f6d35cca602e62
I20240503 14:14:42.884394 18121 catalog_manager.cc:5484] T 10a4ca83469d4dab823a26d6cbe952a6 P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:42.885564 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10a4ca83469d4dab823a26d6cbe952a6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:42.889393 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 10a4ca83469d4dab823a26d6cbe952a6 with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:42.891511 22411 tablet_replica.cc:330] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:42.891770 22411 raft_consensus.cc:2229] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:42.892150 22411 raft_consensus.cc:2258] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:42.893239 22411 ts_tablet_manager.cc:1875] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:42.898104 20209 raft_consensus.cc:1203] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:42.900094 22411 ts_tablet_manager.cc:1888] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:42.900338 22411 log.cc:1196] T 10a4ca83469d4dab823a26d6cbe952a6 P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/10a4ca83469d4dab823a26d6cbe952a6
I20240503 14:14:42.901086 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet 10a4ca83469d4dab823a26d6cbe952a6 (table rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]) successfully deleted
I20240503 14:14:42.932706 22111 tablet_copy_client.cc:523] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:42.935778 22111 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap starting.
I20240503 14:14:42.978658 22111 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:42.979079 22111 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap complete.
I20240503 14:14:42.979362 22111 ts_tablet_manager.cc:1367] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent bootstrapping tablet: real 0.044s user 0.042s sys 0.000s
I20240503 14:14:42.980089 22111 raft_consensus.cc:348] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:42.980404 22111 raft_consensus.cc:729] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e6a3f22c7ae444e96f6d35cca602e62, State: Initialized, Role: LEARNER
I20240503 14:14:42.980615 22111 consensus_queue.cc:260] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:42.981359 22111 ts_tablet_manager.cc:1398] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240503 14:14:42.985817 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-f2c5c6f4317649fab75d01e339b76403 received from {username='slave'} at 127.10.145.5:45257
I20240503 14:14:42.985996 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-f2c5c6f4317649fab75d01e339b76403 on tablet f2c5c6f4317649fab75d01e339b76403 with peer 2e6a3f22c7ae444e96f6d35cca602e62
I20240503 14:14:43.080469 20209 raft_consensus.cc:1203] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:43.179180 22215 tablet_copy_client.cc:523] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:43.182664 22215 tablet_bootstrap.cc:492] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: Bootstrap starting.
I20240503 14:14:43.198235 20341 raft_consensus.cc:1203] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:43.217181 22215 tablet_bootstrap.cc:492] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:43.217643 22215 tablet_bootstrap.cc:492] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: Bootstrap complete.
I20240503 14:14:43.217932 22215 ts_tablet_manager.cc:1367] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: Time spent bootstrapping tablet: real 0.035s user 0.030s sys 0.004s
I20240503 14:14:43.218739 22215 raft_consensus.cc:348] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:43.218973 22215 raft_consensus.cc:729] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Initialized, Role: LEARNER
I20240503 14:14:43.219166 22215 consensus_queue.cc:260] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:43.219756 22215 ts_tablet_manager.cc:1398] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:43.220561 18679 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session 80260781ae364d049c2500ff76a06ce8-3de72df2a82e445f9aa783bc741fa996 received from {username='slave'} at 127.10.145.8:39351
I20240503 14:14:43.220686 18679 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session 80260781ae364d049c2500ff76a06ce8-3de72df2a82e445f9aa783bc741fa996 on tablet 3de72df2a82e445f9aa783bc741fa996 with peer 80260781ae364d049c2500ff76a06ce8
I20240503 14:14:43.224853 20605 raft_consensus.cc:1203] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:43.271927 22221 tablet_copy_client.cc:523] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:43.274907 22221 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap starting.
I20240503 14:14:43.308081 22221 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:43.308504 22221 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap complete.
I20240503 14:14:43.308782 22221 ts_tablet_manager.cc:1367] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.005s
I20240503 14:14:43.309531 22221 raft_consensus.cc:348] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:43.309749 22221 raft_consensus.cc:729] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e6a3f22c7ae444e96f6d35cca602e62, State: Initialized, Role: LEARNER
I20240503 14:14:43.309950 22221 consensus_queue.cc:260] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:43.310415 22221 ts_tablet_manager.cc:1398] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:43.311400 18550 tablet_copy_service.cc:342] P 55339e38e7aa4bb7b7b42aa206108722: Request end of tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-71a326db29114366951a67b9aa84ac03 received from {username='slave'} at 127.10.145.5:59583
I20240503 14:14:43.311563 18550 tablet_copy_service.cc:433] P 55339e38e7aa4bb7b7b42aa206108722: ending tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-71a326db29114366951a67b9aa84ac03 on tablet 71a326db29114366951a67b9aa84ac03 with peer 2e6a3f22c7ae444e96f6d35cca602e62
I20240503 14:14:43.323045 21671 raft_consensus.cc:1050] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER 80260781ae364d049c2500ff76a06ce8 to VOTER
I20240503 14:14:43.323843 21671 consensus_queue.cc:237] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:43.326911 20605 raft_consensus.cc:1261] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.327111 18529 raft_consensus.cc:1261] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.327634 22501 consensus_queue.cc:1035] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.327893 18400 raft_consensus.cc:1261] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.328222 22503 consensus_queue.cc:1035] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.328687 21671 consensus_queue.cc:1035] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.334607 22501 raft_consensus.cc:2932] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.335381 18400 raft_consensus.cc:2932] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.342098 20605 raft_consensus.cc:2932] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.342788 18128 catalog_manager.cc:5484] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.349982 20209 raft_consensus.cc:1203] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:43.354256 18660 consensus_queue.cc:237] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:43.358031 18400 raft_consensus.cc:1261] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.359460 22501 consensus_queue.cc:1035] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.359692 20605 raft_consensus.cc:1261] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.360249 22503 consensus_queue.cc:1035] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.362828 22501 raft_consensus.cc:2932] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.363488 20605 raft_consensus.cc:2932] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.364645 18400 raft_consensus.cc:2932] T f09411ea6c414451b66fd457d9aaf010 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.366770 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f09411ea6c414451b66fd457d9aaf010 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:43.368921 18120 catalog_manager.cc:5484] T f09411ea6c414451b66fd457d9aaf010 P 80260781ae364d049c2500ff76a06ce8 reported cstate change: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.373883 20918 raft_consensus.cc:1050] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER a632094ba4554a4895d6c0b538b81af8 to VOTER
I20240503 14:14:43.374786 20918 consensus_queue.cc:237] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:43.375046 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet f09411ea6c414451b66fd457d9aaf010 with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:43.377072 22512 tablet_replica.cc:330] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:43.377357 22512 raft_consensus.cc:2229] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:43.378831 22512 pending_rounds.cc:62] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: Trying to abort 1 pending ops.
I20240503 14:14:43.379112 18268 raft_consensus.cc:1261] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.379443 18400 raft_consensus.cc:1261] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.379740 22501 consensus_queue.cc:1035] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.379118 22512 pending_rounds.cc:68] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023598318892765184 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f09411ea6c414451b66fd457d9aaf010" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } } }
I20240503 14:14:43.380100 21671 consensus_queue.cc:1035] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.380239 22512 raft_consensus.cc:2894] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 14:14:43.380369 22512 raft_consensus.cc:2866] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:14:43.380493 22512 raft_consensus.cc:2258] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:43.381839 22512 ts_tablet_manager.cc:1875] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:43.382301 20475 raft_consensus.cc:1261] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.382895 22501 consensus_queue.cc:1035] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.386168 21671 raft_consensus.cc:2932] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:43.386744 20475 raft_consensus.cc:2932] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:43.392117 18268 raft_consensus.cc:2932] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:43.393611 22512 ts_tablet_manager.cc:1888] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:43.393792 22512 log.cc:1196] T f09411ea6c414451b66fd457d9aaf010 P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/f09411ea6c414451b66fd457d9aaf010
I20240503 14:14:43.394863 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet f09411ea6c414451b66fd457d9aaf010 (table rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]) successfully deleted
I20240503 14:14:43.397691 18400 raft_consensus.cc:2932] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:43.404084 18120 catalog_manager.cc:5484] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8 reported cstate change: config changed from index 4 to 5, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:43.412592 18660 consensus_queue.cc:237] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:43.415249 20475 raft_consensus.cc:1261] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.415266 18400 raft_consensus.cc:1261] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.415942 22501 consensus_queue.cc:1035] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.416285 20918 consensus_queue.cc:1035] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.420137 21671 raft_consensus.cc:2932] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:43.420884 20475 raft_consensus.cc:2932] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:43.424528 18400 raft_consensus.cc:2932] T c2167226099a4dcf98304918e8f412af P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:43.426070 18119 catalog_manager.cc:5484] T c2167226099a4dcf98304918e8f412af P a632094ba4554a4895d6c0b538b81af8 reported cstate change: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:43.427803 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2167226099a4dcf98304918e8f412af with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:43.430776 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet c2167226099a4dcf98304918e8f412af with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:43.431605 22529 tablet_replica.cc:330] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:43.431939 22529 raft_consensus.cc:2229] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:43.432659 22529 raft_consensus.cc:2258] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:43.434048 22529 ts_tablet_manager.cc:1875] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:43.435986 21420 raft_consensus.cc:1050] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 to VOTER
I20240503 14:14:43.436887 21420 consensus_queue.cc:237] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:43.441709 22529 ts_tablet_manager.cc:1888] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:43.441967 22529 log.cc:1196] T c2167226099a4dcf98304918e8f412af P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/c2167226099a4dcf98304918e8f412af
I20240503 14:14:43.443162 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet c2167226099a4dcf98304918e8f412af (table rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]) successfully deleted
I20240503 14:14:43.444319 20209 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.444769 18400 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.445379 18529 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.445401 21414 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.446044 21414 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.446595 21414 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.451645 18529 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.452498 21414 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.457499 20209 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.462455 18128 catalog_manager.cc:5484] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:43.463938 18400 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.467967 18268 consensus_queue.cc:237] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:43.470240 20209 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.470309 18529 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.471171 21414 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.472764 21414 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.474364 21414 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.478276 18529 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.479084 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe3d20ad69474bcbae19bdd06cbfeb52 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:43.482565 18119 catalog_manager.cc:5484] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:43.484030 20209 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.490517 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3d20ad69474bcbae19bdd06cbfeb52 with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:43.497903 22545 tablet_replica.cc:330] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:43.498175 22545 raft_consensus.cc:2229] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:43.498667 22545 raft_consensus.cc:2258] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:43.499778 22545 ts_tablet_manager.cc:1875] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:43.511709 22545 ts_tablet_manager.cc:1888] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:43.511907 22545 log.cc:1196] T fe3d20ad69474bcbae19bdd06cbfeb52 P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe3d20ad69474bcbae19bdd06cbfeb52
I20240503 14:14:43.512954 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet fe3d20ad69474bcbae19bdd06cbfeb52 (table rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]) successfully deleted
I20240503 14:14:43.549137 22184 raft_consensus.cc:1050] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568: attempting to promote NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 to VOTER
I20240503 14:14:43.550231 22184 consensus_queue.cc:237] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:43.553504 20209 raft_consensus.cc:1261] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.553571 18529 raft_consensus.cc:1261] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.553876 18660 raft_consensus.cc:1261] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.554746 22568 consensus_queue.cc:1035] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.555292 22478 consensus_queue.cc:1035] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.556389 22478 consensus_queue.cc:1035] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.569805 22568 raft_consensus.cc:2932] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.570976 18529 raft_consensus.cc:2932] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.571010 20209 raft_consensus.cc:2932] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.574738 18660 raft_consensus.cc:2932] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.579947 18121 catalog_manager.cc:5484] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62 reported cstate change: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.587736 18400 consensus_queue.cc:237] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:43.590732 18660 raft_consensus.cc:1261] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.591444 22568 consensus_queue.cc:1035] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.592145 20209 raft_consensus.cc:1261] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.594455 22561 consensus_queue.cc:1035] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.595331 22478 raft_consensus.cc:2932] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.596027 18660 raft_consensus.cc:2932] T d5b74777c90f4cb7937d0dfc54420467 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.600581 18114 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5b74777c90f4cb7937d0dfc54420467 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:43.601528 18119 catalog_manager.cc:5484] T d5b74777c90f4cb7937d0dfc54420467 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:43.601935 20209 raft_consensus.cc:2932] T d5b74777c90f4cb7937d0dfc54420467 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.606379 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b74777c90f4cb7937d0dfc54420467 with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:43.607517 22512 tablet_replica.cc:330] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:43.607764 22512 raft_consensus.cc:2229] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:43.608065 22512 raft_consensus.cc:2258] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:43.609504 22512 ts_tablet_manager.cc:1875] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:43.613416 22535 raft_consensus.cc:1050] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 80260781ae364d049c2500ff76a06ce8 to VOTER
I20240503 14:14:43.614465 22535 consensus_queue.cc:237] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:43.616132 22512 ts_tablet_manager.cc:1888] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:43.616349 22512 log.cc:1196] T d5b74777c90f4cb7937d0dfc54420467 P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/d5b74777c90f4cb7937d0dfc54420467
I20240503 14:14:43.617251 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet d5b74777c90f4cb7937d0dfc54420467 (table rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]) successfully deleted
I20240503 14:14:43.618777 18660 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.619029 18529 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.619405 20605 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.619457 21420 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.619961 21414 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.620290 22535 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.637588 21414 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.638494 18529 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.642161 18660 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.643807 18119 catalog_manager.cc:5484] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:43.648577 20605 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.654712 18268 consensus_queue.cc:237] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:43.656816 20605 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.657001 18529 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.657429 21420 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.657831 21414 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.660028 21414 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.660784 20605 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.664256 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4cb29bdb71f24b46b32e7b97d0172c20 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:43.665228 18529 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:43.666473 18119 catalog_manager.cc:5484] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 5 to 6, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:43.671967 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb29bdb71f24b46b32e7b97d0172c20 with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:43.672753 22596 tablet_replica.cc:330] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:43.672969 22596 raft_consensus.cc:2229] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:43.673418 22596 raft_consensus.cc:2258] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:43.674588 22596 ts_tablet_manager.cc:1875] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:43.680732 22596 ts_tablet_manager.cc:1888] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:43.680881 22596 log.cc:1196] T 4cb29bdb71f24b46b32e7b97d0172c20 P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/4cb29bdb71f24b46b32e7b97d0172c20
I20240503 14:14:43.682044 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet 4cb29bdb71f24b46b32e7b97d0172c20 (table rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]) successfully deleted
I20240503 14:14:43.717643 20605 raft_consensus.cc:1203] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:43.755649 20918 raft_consensus.cc:1050] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 to VOTER
I20240503 14:14:43.756398 20918 consensus_queue.cc:237] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:43.758780 20341 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.758832 18400 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.758863 18268 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.759404 20918 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.759752 21671 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.760043 22503 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.763794 22536 raft_consensus.cc:1050] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 to VOTER
I20240503 14:14:43.764676 22536 consensus_queue.cc:237] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:43.766736 21671 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:43.767716 18400 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.767771 18529 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.767968 20209 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:43.768301 22536 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.768805 22536 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.769407 21414 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:43.770206 20341 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:43.772843 18398 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:43.776301 18128 catalog_manager.cc:5484] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 4 to 5, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:43.781509 20210 raft_consensus.cc:1203] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:43.788854 18660 consensus_queue.cc:237] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:43.791253 20341 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.791254 18398 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.791862 22503 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.793660 22503 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.794157 21414 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.796768 22503 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:43.799468 18398 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.799480 18399 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:43.803422 18529 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.803980 18120 catalog_manager.cc:5484] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:43.804728 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5bc67ebb58a246d9a8c47f0838a4831a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:43.807360 18119 catalog_manager.cc:5484] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:43.807439 20342 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:43.811024 20209 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.811695 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc67ebb58a246d9a8c47f0838a4831a with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:43.812311 22529 tablet_replica.cc:330] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:43.812536 22529 raft_consensus.cc:2229] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:43.812724 22529 pending_rounds.cc:62] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: Trying to abort 1 pending ops.
I20240503 14:14:43.812860 22529 pending_rounds.cc:68] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023598320664420352 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5bc67ebb58a246d9a8c47f0838a4831a" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } } }
I20240503 14:14:43.813362 22529 raft_consensus.cc:2894] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 14:14:43.813472 22529 raft_consensus.cc:2866] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:14:43.813560 22529 raft_consensus.cc:2258] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:43.814489 22529 ts_tablet_manager.cc:1875] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:43.821861 22529 ts_tablet_manager.cc:1888] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:43.822064 22529 log.cc:1196] T 5bc67ebb58a246d9a8c47f0838a4831a P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/5bc67ebb58a246d9a8c47f0838a4831a
I20240503 14:14:43.822827 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet 5bc67ebb58a246d9a8c47f0838a4831a (table rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]) successfully deleted
I20240503 14:14:43.825224 18269 consensus_queue.cc:237] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:43.827600 18529 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.828186 21420 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.830873 21414 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.831416 20209 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:43.831470 18529 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.832104 22536 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:43.833896 20209 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.835283 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2c5c6f4317649fab75d01e339b76403 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:43.836941 18121 catalog_manager.cc:5484] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:43.841259 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c5c6f4317649fab75d01e339b76403 with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:43.841717 22545 tablet_replica.cc:330] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:43.841954 22545 raft_consensus.cc:2229] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:43.842216 22545 raft_consensus.cc:2258] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:43.843387 22545 ts_tablet_manager.cc:1875] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:43.848232 22545 ts_tablet_manager.cc:1888] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:43.848373 22545 log.cc:1196] T f2c5c6f4317649fab75d01e339b76403 P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/f2c5c6f4317649fab75d01e339b76403
I20240503 14:14:43.849154 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet f2c5c6f4317649fab75d01e339b76403 (table rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]) successfully deleted
I20240503 14:14:44.241384 22329 raft_consensus.cc:1050] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: attempting to promote NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 to VOTER
I20240503 14:14:44.242226 22329 consensus_queue.cc:237] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:44.244884 20209 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:44.244939 18660 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:44.245613 22671 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:44.246534 18398 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:44.249191 22673 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:14:44.249948 22329 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:44.253496 22329 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:44.254598 18657 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:44.262233 18119 catalog_manager.cc:5484] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:44.265800 20209 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:44.272629 18398 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:44.278621 18529 consensus_queue.cc:237] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:44.282126 20209 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:44.283260 18657 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:44.283454 22678 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:44.284543 22670 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:44.286648 22329 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:44.287276 20209 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:44.288808 18657 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:44.290447 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71a326db29114366951a67b9aa84ac03 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:44.292888 18128 catalog_manager.cc:5484] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 5 to 6, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:44.295025 22503 raft_consensus.cc:1050] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER 80260781ae364d049c2500ff76a06ce8 to VOTER
I20240503 14:14:44.295804 22503 consensus_queue.cc:237] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:44.297744 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a326db29114366951a67b9aa84ac03 with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:44.298111 22545 tablet_replica.cc:330] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:44.298327 22545 raft_consensus.cc:2229] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:44.298614 22545 raft_consensus.cc:2258] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:44.299597 22545 ts_tablet_manager.cc:1875] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:44.300467 20605 raft_consensus.cc:1261] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:44.300755 18529 raft_consensus.cc:1261] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:44.301213 21671 consensus_queue.cc:1035] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:44.302604 21671 consensus_queue.cc:1035] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:44.303965 18399 raft_consensus.cc:1261] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:44.304720 20918 consensus_queue.cc:1035] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:44.307796 22545 ts_tablet_manager.cc:1888] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:44.308046 22545 log.cc:1196] T 71a326db29114366951a67b9aa84ac03 P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/71a326db29114366951a67b9aa84ac03
I20240503 14:14:44.307878 21671 raft_consensus.cc:2932] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:44.308933 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet 71a326db29114366951a67b9aa84ac03 (table rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]) successfully deleted
I20240503 14:14:44.312077 18529 raft_consensus.cc:2932] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:44.313314 18399 raft_consensus.cc:2932] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:44.313462 18121 catalog_manager.cc:5484] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:44.317925 20604 raft_consensus.cc:2932] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:44.326105 18657 consensus_queue.cc:237] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:44.328212 20604 raft_consensus.cc:1261] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:44.328249 18399 raft_consensus.cc:1261] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:14:44.328941 20918 consensus_queue.cc:1035] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:44.329313 21671 consensus_queue.cc:1035] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:14:44.331771 22503 raft_consensus.cc:2932] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:44.332465 18399 raft_consensus.cc:2932] T 3de72df2a82e445f9aa783bc741fa996 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:44.332705 20604 raft_consensus.cc:2932] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:44.334978 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3de72df2a82e445f9aa783bc741fa996 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:44.337252 18121 catalog_manager.cc:5484] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 5 to 6, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:44.341702 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet 3de72df2a82e445f9aa783bc741fa996 with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:44.342687 22683 tablet_replica.cc:330] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:44.342921 22683 raft_consensus.cc:2229] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:44.343184 22683 raft_consensus.cc:2258] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:44.344007 22683 ts_tablet_manager.cc:1875] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:44.349092 22683 ts_tablet_manager.cc:1888] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:14:44.349236 22683 log.cc:1196] T 3de72df2a82e445f9aa783bc741fa996 P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/3de72df2a82e445f9aa783bc741fa996
I20240503 14:14:44.350090 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet 3de72df2a82e445f9aa783bc741fa996 (table rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]) successfully deleted
I20240503 14:14:44.598106 18639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:44.630298 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:44.639477 18510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:44.643471 20322 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:44.663558 20585 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:44.663642 18248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:44.697441 18378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:44.704782 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:44.720960 21521 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:44.813627 20604 consensus_queue.cc:237] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:44.818588 18399 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:44.819465 21864 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:44.821031 18657 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:44.821918 21864 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:44.823796 21864 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:44.824357 18399 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
W20240503 14:14:44.826395 20541 consensus_peers.cc:485] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 -> Peer c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329): Couldn't send request to peer c36ba962bcb34dbea55f4e55689bd38a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91743575cf4a4f38b5b43265539237a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:44.830243 18128 catalog_manager.cc:5484] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 7 to 8, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New cstate: current_term: 2 leader_uuid: "80260781ae364d049c2500ff76a06ce8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:44.842487 18657 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:44.858952 18657 consensus_queue.cc:237] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:44.861547 18399 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:44.862213 20209 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 14:14:44.862649 18596 consensus_peers.cc:485] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 -> Peer a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307): Couldn't send request to peer a632094ba4554a4895d6c0b538b81af8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79373bd79a934eaab67bbcf2208e7b5b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:44.863392 21671 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:44.864379 20918 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:44.866686 21671 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:44.870846 20210 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:44.872094 18399 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:44.872815 18120 catalog_manager.cc:5484] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New cstate: current_term: 2 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:44.892952 18528 consensus_queue.cc:237] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:44.895182 18657 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:44.895627 18464 consensus_peers.cc:485] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 -> Peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Couldn't send request to peer 69cbbbedfe03437b8c95b7bb730dbe66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71a326db29114366951a67b9aa84ac03. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:44.895794 22744 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:44.896617 20210 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:44.897199 22744 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:44.899551 22744 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:44.900107 18657 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:44.904806 20210 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:44.910231 18128 catalog_manager.cc:5484] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:44.949498 18269 consensus_queue.cc:237] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:44.952500 18399 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:44.953205 20210 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:44.953315 22775 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:44.953847 21414 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:44.957031 21414 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:44.957610 18399 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
W20240503 14:14:44.961905 18205 consensus_peers.cc:485] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 -> Peer c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329): Couldn't send request to peer c36ba962bcb34dbea55f4e55689bd38a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eeb40a038c4d404d9cb8ea693d9071d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:44.963667 20210 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:44.964164 18120 catalog_manager.cc:5484] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:44.979511 18399 consensus_queue.cc:237] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
W20240503 14:14:44.982213 18333 consensus_peers.cc:485] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 -> Peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107): Couldn't send request to peer 80260781ae364d049c2500ff76a06ce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 963ecc30d66b43a4ad149f596bd288d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:44.982446 20341 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:44.983191 22786 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:44.986020 18657 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:44.986181 22568 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:44.987010 20341 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:44.986920 22788 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:44.995160 18120 catalog_manager.cc:5484] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 6 to 7, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:45.010787 18657 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:45.040387 18528 consensus_queue.cc:237] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:45.042932 18657 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.042968 20341 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:45.043056 18466 consensus_peers.cc:485] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 -> Peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Couldn't send request to peer 2e6a3f22c7ae444e96f6d35cca602e62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19b1b6be3bda47829f53765282eae9f9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.043574 22744 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.043982 22767 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.046639 22744 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:45.047353 20341 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:45.048058 18657 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } } }
I20240503 14:14:45.052680 18120 catalog_manager.cc:5484] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 6 to 7, NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) added. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:45.070466 18269 consensus_queue.cc:237] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:45.072786 20210 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:45.072901 18203 consensus_peers.cc:485] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 -> Peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Couldn't send request to peer 69cbbbedfe03437b8c95b7bb730dbe66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eedc3cfb2dff4f9084446d89b9b3c0f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.073192 18399 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.073380 22775 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.073817 22779 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.076205 22775 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:45.076849 20210 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:45.077335 18399 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:45.084836 18121 catalog_manager.cc:5484] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:45.112675 20604 consensus_queue.cc:237] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:45.116019 18657 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:45.116108 18528 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:45.116672 22750 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:45.117144 21864 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:14:45.124969 20541 consensus_peers.cc:485] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 -> Peer a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307): Couldn't send request to peer a632094ba4554a4895d6c0b538b81af8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea3ea2e3d87f4c099abbf5193c239a8b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.127171 21864 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:45.128005 18528 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:45.131145 18660 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:45.134637 18128 catalog_manager.cc:5484] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New cstate: current_term: 2 leader_uuid: "80260781ae364d049c2500ff76a06ce8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:45.155586 18528 consensus_queue.cc:237] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:45.159026 18269 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.159963 20475 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.160451 22678 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.161121 22678 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.174813 22744 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
W20240503 14:14:45.176563 18466 consensus_peers.cc:485] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 -> Peer c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329): Couldn't send request to peer c36ba962bcb34dbea55f4e55689bd38a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f9705f607a94835ad7e6c20668b80ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.177349 18269 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:45.179044 20475 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:45.184521 18120 catalog_manager.cc:5484] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:45.217361 18528 consensus_queue.cc:237] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:45.220165 20475 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.220523 18657 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.220892 22744 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:14:45.221246 18463 consensus_peers.cc:485] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 -> Peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107): Couldn't send request to peer 80260781ae364d049c2500ff76a06ce8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c63b35eef494f0e8928c4ca0f74ec32. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.221470 22767 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.224217 22744 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:45.224931 20475 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:45.230209 18120 catalog_manager.cc:5484] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 6 to 7, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:45.233448 18657 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } } }
I20240503 14:14:45.250984 18399 consensus_queue.cc:237] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:45.253885 20604 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.254069 18269 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.254566 22786 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.255681 22786 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.272511 22786 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
W20240503 14:14:45.273190 18336 consensus_peers.cc:485] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 -> Peer c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329): Couldn't send request to peer c36ba962bcb34dbea55f4e55689bd38a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c337ef0405e4856bd2a608a6a602a49. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.279273 18128 catalog_manager.cc:5484] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:45.280189 18269 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:45.280586 20604 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:45.304288 18269 consensus_queue.cc:237] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:45.307027 20604 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.307097 18528 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:45.307356 18203 consensus_peers.cc:485] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 -> Peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Couldn't send request to peer 69cbbbedfe03437b8c95b7bb730dbe66. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cb29bdb71f24b46b32e7b97d0172c20. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.307770 22808 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.308058 22775 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.311589 21414 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:45.319222 20604 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:45.323928 22852 ts_tablet_manager.cc:902] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: Initiating tablet copy from peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:45.328433 18121 catalog_manager.cc:5484] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 reported cstate change: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:45.340989 18528 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } } }
I20240503 14:14:45.341814 22852 tablet_copy_client.cc:308] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:38857
I20240503 14:14:45.344604 18550 tablet_copy_service.cc:140] P 55339e38e7aa4bb7b7b42aa206108722: Received BeginTabletCopySession request for tablet 71a326db29114366951a67b9aa84ac03 from peer 69cbbbedfe03437b8c95b7bb730dbe66 ({username='slave'} at 127.10.145.6:38759)
I20240503 14:14:45.344810 18550 tablet_copy_service.cc:161] P 55339e38e7aa4bb7b7b42aa206108722: Beginning new tablet copy session on tablet 71a326db29114366951a67b9aa84ac03 from peer 69cbbbedfe03437b8c95b7bb730dbe66 at {username='slave'} at 127.10.145.6:38759: session id = 69cbbbedfe03437b8c95b7bb730dbe66-71a326db29114366951a67b9aa84ac03
I20240503 14:14:45.351171 18550 tablet_copy_source_session.cc:205] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.356943 18657 consensus_queue.cc:237] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:45.357220 22852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a326db29114366951a67b9aa84ac03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.361861 20605 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.362646 22758 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.374094 18528 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.373984 22758 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:45.377413 21671 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.379261 20605 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:45.385556 18121 catalog_manager.cc:5484] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:45.385953 22866 ts_tablet_manager.cc:902] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:45.397707 22867 ts_tablet_manager.cc:902] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: Initiating tablet copy from peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107)
I20240503 14:14:45.401080 18528 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:45.402447 22863 ts_tablet_manager.cc:902] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: Initiating tablet copy from peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:45.403704 22852 tablet_copy_client.cc:783] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.404034 22852 tablet_copy_client.cc:647] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 1 WAL segments...
W20240503 14:14:45.405450 18596 consensus_peers.cc:485] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 -> Peer c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329): Couldn't send request to peer c36ba962bcb34dbea55f4e55689bd38a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7570d5bce0e5481d8dded653617be5a7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.407539 22861 tablet_copy_client.cc:1002] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.414330 22866 tablet_copy_client.cc:308] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:45.421487 22867 tablet_copy_client.cc:308] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.8:44107
I20240503 14:14:45.431726 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet eeb40a038c4d404d9cb8ea693d9071d7 from peer c36ba962bcb34dbea55f4e55689bd38a ({username='slave'} at 127.10.145.9:39445)
I20240503 14:14:45.431962 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet eeb40a038c4d404d9cb8ea693d9071d7 from peer c36ba962bcb34dbea55f4e55689bd38a at {username='slave'} at 127.10.145.9:39445: session id = c36ba962bcb34dbea55f4e55689bd38a-eeb40a038c4d404d9cb8ea693d9071d7
I20240503 14:14:45.434986 18289 tablet_copy_source_session.cc:205] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.437834 22863 tablet_copy_client.cc:308] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:43191
I20240503 14:14:45.438802 18420 tablet_copy_service.cc:140] P ea366e6e15a24c52b1ea5a154f942568: Received BeginTabletCopySession request for tablet 963ecc30d66b43a4ad149f596bd288d9 from peer 80260781ae364d049c2500ff76a06ce8 ({username='slave'} at 127.10.145.8:41697)
I20240503 14:14:45.438998 18420 tablet_copy_service.cc:161] P ea366e6e15a24c52b1ea5a154f942568: Beginning new tablet copy session on tablet 963ecc30d66b43a4ad149f596bd288d9 from peer 80260781ae364d049c2500ff76a06ce8 at {username='slave'} at 127.10.145.8:41697: session id = 80260781ae364d049c2500ff76a06ce8-963ecc30d66b43a4ad149f596bd288d9
I20240503 14:14:45.440248 20625 tablet_copy_service.cc:140] P 80260781ae364d049c2500ff76a06ce8: Received BeginTabletCopySession request for tablet 91743575cf4a4f38b5b43265539237a3 from peer c36ba962bcb34dbea55f4e55689bd38a ({username='slave'} at 127.10.145.9:54867)
I20240503 14:14:45.440469 20625 tablet_copy_service.cc:161] P 80260781ae364d049c2500ff76a06ce8: Beginning new tablet copy session on tablet 91743575cf4a4f38b5b43265539237a3 from peer c36ba962bcb34dbea55f4e55689bd38a at {username='slave'} at 127.10.145.9:54867: session id = c36ba962bcb34dbea55f4e55689bd38a-91743575cf4a4f38b5b43265539237a3
I20240503 14:14:45.442039 22866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb40a038c4d404d9cb8ea693d9071d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.442167 18420 tablet_copy_source_session.cc:205] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.443763 22863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 963ecc30d66b43a4ad149f596bd288d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.443996 20625 tablet_copy_source_session.cc:205] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.445925 22867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91743575cf4a4f38b5b43265539237a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.448215 20341 consensus_queue.cc:237] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:45.451337 18528 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:45.457577 22866 tablet_copy_client.cc:783] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.457739 22867 tablet_copy_client.cc:783] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.457963 22866 tablet_copy_client.cc:647] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.458071 22867 tablet_copy_client.cc:647] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.460613 22878 tablet_copy_client.cc:1002] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.460798 22880 tablet_copy_client.cc:1002] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.464115 22863 tablet_copy_client.cc:783] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.465498 22863 tablet_copy_client.cc:647] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.467386 22888 tablet_copy_client.cc:1002] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.469880 21277 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:45.471083 22892 ts_tablet_manager.cc:902] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:45.473781 18269 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:45.476178 22890 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:45.491286 22890 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:45.494863 18268 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:45.500334 18528 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:45.501825 22893 ts_tablet_manager.cc:902] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: Initiating tablet copy from peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:45.507709 18121 catalog_manager.cc:5484] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 7 to 8, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New cstate: current_term: 2 leader_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
W20240503 14:14:45.514480 20278 consensus_peers.cc:485] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 -> Peer a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307): Couldn't send request to peer a632094ba4554a4895d6c0b538b81af8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0c1a7f5739f4fa59271254e501fb095. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.538750 22892 tablet_copy_client.cc:308] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:45.539805 18679 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet 79373bd79a934eaab67bbcf2208e7b5b from peer a632094ba4554a4895d6c0b538b81af8 ({username='slave'} at 127.10.145.7:47505)
I20240503 14:14:45.540010 18679 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet 79373bd79a934eaab67bbcf2208e7b5b from peer a632094ba4554a4895d6c0b538b81af8 at {username='slave'} at 127.10.145.7:47505: session id = a632094ba4554a4895d6c0b538b81af8-79373bd79a934eaab67bbcf2208e7b5b
I20240503 14:14:45.542884 22893 tablet_copy_client.cc:308] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:38857
I20240503 14:14:45.543323 18679 tablet_copy_source_session.cc:205] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.543749 18550 tablet_copy_service.cc:140] P 55339e38e7aa4bb7b7b42aa206108722: Received BeginTabletCopySession request for tablet 19b1b6be3bda47829f53765282eae9f9 from peer 2e6a3f22c7ae444e96f6d35cca602e62 ({username='slave'} at 127.10.145.5:59583)
I20240503 14:14:45.543911 18550 tablet_copy_service.cc:161] P 55339e38e7aa4bb7b7b42aa206108722: Beginning new tablet copy session on tablet 19b1b6be3bda47829f53765282eae9f9 from peer 2e6a3f22c7ae444e96f6d35cca602e62 at {username='slave'} at 127.10.145.5:59583: session id = 2e6a3f22c7ae444e96f6d35cca602e62-19b1b6be3bda47829f53765282eae9f9
I20240503 14:14:45.545441 22892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79373bd79a934eaab67bbcf2208e7b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.547160 18550 tablet_copy_source_session.cc:205] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.548980 22893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b1b6be3bda47829f53765282eae9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.556229 22893 tablet_copy_client.cc:783] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.556888 22893 tablet_copy_client.cc:647] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.558357 22892 tablet_copy_client.cc:783] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.558663 22892 tablet_copy_client.cc:647] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.559082 22904 tablet_copy_client.cc:1002] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.562845 22903 tablet_copy_client.cc:1002] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.578485 18657 consensus_queue.cc:237] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:45.581022 18529 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:45.581035 20210 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:45.581313 18596 consensus_peers.cc:485] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 -> Peer a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307): Couldn't send request to peer a632094ba4554a4895d6c0b538b81af8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9577e6d3801f42f89afe8a48a77031dd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:45.581656 22758 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.582037 21671 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:45.584848 21671 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:45.585506 18529 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:45.590132 20210 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } } }
I20240503 14:14:45.592448 18128 catalog_manager.cc:5484] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 6 to 7, NON_VOTER a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) added. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:45.600924 22919 ts_tablet_manager.cc:902] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: Initiating tablet copy from peer 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107)
I20240503 14:14:45.609261 22928 ts_tablet_manager.cc:902] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:45.610044 22919 tablet_copy_client.cc:308] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.8:44107
I20240503 14:14:45.621392 22928 tablet_copy_client.cc:308] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:45.622164 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet eedc3cfb2dff4f9084446d89b9b3c0f4 from peer 69cbbbedfe03437b8c95b7bb730dbe66 ({username='slave'} at 127.10.145.6:41609)
I20240503 14:14:45.622319 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet eedc3cfb2dff4f9084446d89b9b3c0f4 from peer 69cbbbedfe03437b8c95b7bb730dbe66 at {username='slave'} at 127.10.145.6:41609: session id = 69cbbbedfe03437b8c95b7bb730dbe66-eedc3cfb2dff4f9084446d89b9b3c0f4
I20240503 14:14:45.622380 20625 tablet_copy_service.cc:140] P 80260781ae364d049c2500ff76a06ce8: Received BeginTabletCopySession request for tablet ea3ea2e3d87f4c099abbf5193c239a8b from peer a632094ba4554a4895d6c0b538b81af8 ({username='slave'} at 127.10.145.7:48145)
I20240503 14:14:45.622604 20625 tablet_copy_service.cc:161] P 80260781ae364d049c2500ff76a06ce8: Beginning new tablet copy session on tablet ea3ea2e3d87f4c099abbf5193c239a8b from peer a632094ba4554a4895d6c0b538b81af8 at {username='slave'} at 127.10.145.7:48145: session id = a632094ba4554a4895d6c0b538b81af8-ea3ea2e3d87f4c099abbf5193c239a8b
I20240503 14:14:45.624447 18289 tablet_copy_source_session.cc:205] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.625468 20625 tablet_copy_source_session.cc:205] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.625895 22928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eedc3cfb2dff4f9084446d89b9b3c0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.626806 22919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3ea2e3d87f4c099abbf5193c239a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.631953 22928 tablet_copy_client.cc:783] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.632302 22928 tablet_copy_client.cc:647] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.634274 22934 tablet_copy_client.cc:1002] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.634565 22919 tablet_copy_client.cc:783] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.634842 22919 tablet_copy_client.cc:647] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.636541 22929 tablet_copy_client.cc:1002] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.658869 22955 ts_tablet_manager.cc:902] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: Initiating tablet copy from peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:45.661432 22955 tablet_copy_client.cc:308] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:38857
I20240503 14:14:45.662232 18550 tablet_copy_service.cc:140] P 55339e38e7aa4bb7b7b42aa206108722: Received BeginTabletCopySession request for tablet 7c63b35eef494f0e8928c4ca0f74ec32 from peer 80260781ae364d049c2500ff76a06ce8 ({username='slave'} at 127.10.145.8:40531)
I20240503 14:14:45.662426 18550 tablet_copy_service.cc:161] P 55339e38e7aa4bb7b7b42aa206108722: Beginning new tablet copy session on tablet 7c63b35eef494f0e8928c4ca0f74ec32 from peer 80260781ae364d049c2500ff76a06ce8 at {username='slave'} at 127.10.145.8:40531: session id = 80260781ae364d049c2500ff76a06ce8-7c63b35eef494f0e8928c4ca0f74ec32
I20240503 14:14:45.665305 18550 tablet_copy_source_session.cc:205] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.666725 22955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c63b35eef494f0e8928c4ca0f74ec32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.672796 22955 tablet_copy_client.cc:783] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.673074 22955 tablet_copy_client.cc:647] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.675094 22959 tablet_copy_client.cc:1002] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.704916 22985 ts_tablet_manager.cc:902] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:45.708550 22985 tablet_copy_client.cc:308] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:45.709410 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet 4cb29bdb71f24b46b32e7b97d0172c20 from peer 69cbbbedfe03437b8c95b7bb730dbe66 ({username='slave'} at 127.10.145.6:41609)
I20240503 14:14:45.709616 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet 4cb29bdb71f24b46b32e7b97d0172c20 from peer 69cbbbedfe03437b8c95b7bb730dbe66 at {username='slave'} at 127.10.145.6:41609: session id = 69cbbbedfe03437b8c95b7bb730dbe66-4cb29bdb71f24b46b32e7b97d0172c20
I20240503 14:14:45.712241 18289 tablet_copy_source_session.cc:205] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.713774 22985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb29bdb71f24b46b32e7b97d0172c20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.719538 22985 tablet_copy_client.cc:783] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.719792 22985 tablet_copy_client.cc:647] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.721747 22989 tablet_copy_client.cc:1002] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.728683 23000 ts_tablet_manager.cc:902] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: Initiating tablet copy from peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:45.730660 23000 tablet_copy_client.cc:308] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:38857
I20240503 14:14:45.737124 18550 tablet_copy_service.cc:140] P 55339e38e7aa4bb7b7b42aa206108722: Received BeginTabletCopySession request for tablet 7f9705f607a94835ad7e6c20668b80ad from peer c36ba962bcb34dbea55f4e55689bd38a ({username='slave'} at 127.10.145.9:56393)
I20240503 14:14:45.737372 18550 tablet_copy_service.cc:161] P 55339e38e7aa4bb7b7b42aa206108722: Beginning new tablet copy session on tablet 7f9705f607a94835ad7e6c20668b80ad from peer c36ba962bcb34dbea55f4e55689bd38a at {username='slave'} at 127.10.145.9:56393: session id = c36ba962bcb34dbea55f4e55689bd38a-7f9705f607a94835ad7e6c20668b80ad
I20240503 14:14:45.740032 18550 tablet_copy_source_session.cc:205] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.742462 23000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f9705f607a94835ad7e6c20668b80ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.751371 23000 tablet_copy_client.cc:783] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.751660 23000 tablet_copy_client.cc:647] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.753477 23004 tablet_copy_client.cc:1002] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.778194 23015 ts_tablet_manager.cc:902] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: Initiating tablet copy from peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
I20240503 14:14:45.780130 23015 tablet_copy_client.cc:308] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:43191
I20240503 14:14:45.785784 18420 tablet_copy_service.cc:140] P ea366e6e15a24c52b1ea5a154f942568: Received BeginTabletCopySession request for tablet 8c337ef0405e4856bd2a608a6a602a49 from peer c36ba962bcb34dbea55f4e55689bd38a ({username='slave'} at 127.10.145.9:39915)
I20240503 14:14:45.785945 18420 tablet_copy_service.cc:161] P ea366e6e15a24c52b1ea5a154f942568: Beginning new tablet copy session on tablet 8c337ef0405e4856bd2a608a6a602a49 from peer c36ba962bcb34dbea55f4e55689bd38a at {username='slave'} at 127.10.145.9:39915: session id = c36ba962bcb34dbea55f4e55689bd38a-8c337ef0405e4856bd2a608a6a602a49
I20240503 14:14:45.787858 18420 tablet_copy_source_session.cc:205] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.789207 23015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c337ef0405e4856bd2a608a6a602a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.794011 23015 tablet_copy_client.cc:783] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.794258 23015 tablet_copy_client.cc:647] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.796047 23019 tablet_copy_client.cc:1002] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.867981 23022 ts_tablet_manager.cc:902] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:45.869333 23022 tablet_copy_client.cc:308] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:45.875078 18679 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet 7570d5bce0e5481d8dded653617be5a7 from peer c36ba962bcb34dbea55f4e55689bd38a ({username='slave'} at 127.10.145.9:51127)
I20240503 14:14:45.875276 18679 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet 7570d5bce0e5481d8dded653617be5a7 from peer c36ba962bcb34dbea55f4e55689bd38a at {username='slave'} at 127.10.145.9:51127: session id = c36ba962bcb34dbea55f4e55689bd38a-7570d5bce0e5481d8dded653617be5a7
I20240503 14:14:45.877936 18679 tablet_copy_source_session.cc:205] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:45.879397 23022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7570d5bce0e5481d8dded653617be5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:45.885057 23022 tablet_copy_client.cc:783] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:45.885385 23022 tablet_copy_client.cc:647] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:45.888295 23026 tablet_copy_client.cc:1002] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:45.986292 23044 ts_tablet_manager.cc:902] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:46.005815 23044 tablet_copy_client.cc:308] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:46.009371 23050 ts_tablet_manager.cc:902] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: Initiating tablet copy from peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735)
I20240503 14:14:46.009317 18679 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet 9577e6d3801f42f89afe8a48a77031dd from peer a632094ba4554a4895d6c0b538b81af8 ({username='slave'} at 127.10.145.7:47505)
I20240503 14:14:46.009511 18679 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet 9577e6d3801f42f89afe8a48a77031dd from peer a632094ba4554a4895d6c0b538b81af8 at {username='slave'} at 127.10.145.7:47505: session id = a632094ba4554a4895d6c0b538b81af8-9577e6d3801f42f89afe8a48a77031dd
I20240503 14:14:46.012660 18679 tablet_copy_source_session.cc:205] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:46.014492 23044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9577e6d3801f42f89afe8a48a77031dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:46.020928 23044 tablet_copy_client.cc:783] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:46.021245 23044 tablet_copy_client.cc:647] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:46.023197 23049 tablet_copy_client.cc:1002] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:46.035605 23050 tablet_copy_client.cc:308] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36735
I20240503 14:14:46.043503 20362 tablet_copy_service.cc:140] P 69cbbbedfe03437b8c95b7bb730dbe66: Received BeginTabletCopySession request for tablet f0c1a7f5739f4fa59271254e501fb095 from peer a632094ba4554a4895d6c0b538b81af8 ({username='slave'} at 127.10.145.7:48885)
I20240503 14:14:46.043758 20362 tablet_copy_service.cc:161] P 69cbbbedfe03437b8c95b7bb730dbe66: Beginning new tablet copy session on tablet f0c1a7f5739f4fa59271254e501fb095 from peer a632094ba4554a4895d6c0b538b81af8 at {username='slave'} at 127.10.145.7:48885: session id = a632094ba4554a4895d6c0b538b81af8-f0c1a7f5739f4fa59271254e501fb095
I20240503 14:14:46.047194 20362 tablet_copy_source_session.cc:205] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:46.049182 23050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c1a7f5739f4fa59271254e501fb095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:46.054320 23050 tablet_copy_client.cc:783] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:46.054567 23050 tablet_copy_client.cc:647] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:46.056471 23055 tablet_copy_client.cc:1002] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:46.908026 22852 tablet_copy_client.cc:523] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:46.911553 22852 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap starting.
I20240503 14:14:46.952699 22852 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:46.953110 22852 tablet_bootstrap.cc:492] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap complete.
I20240503 14:14:46.953388 22852 ts_tablet_manager.cc:1367] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent bootstrapping tablet: real 0.042s user 0.043s sys 0.000s
I20240503 14:14:46.954061 22852 raft_consensus.cc:348] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:46.954293 22852 raft_consensus.cc:729] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Initialized, Role: LEARNER
I20240503 14:14:46.954450 22852 consensus_queue.cc:260] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:46.954975 22852 ts_tablet_manager.cc:1398] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:14:46.955818 18550 tablet_copy_service.cc:342] P 55339e38e7aa4bb7b7b42aa206108722: Request end of tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-71a326db29114366951a67b9aa84ac03 received from {username='slave'} at 127.10.145.6:38759
I20240503 14:14:46.955950 18550 tablet_copy_service.cc:433] P 55339e38e7aa4bb7b7b42aa206108722: ending tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-71a326db29114366951a67b9aa84ac03 on tablet 71a326db29114366951a67b9aa84ac03 with peer 69cbbbedfe03437b8c95b7bb730dbe66
I20240503 14:14:46.961211 22867 tablet_copy_client.cc:523] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:46.961205 22866 tablet_copy_client.cc:523] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:46.965442 22866 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap starting.
I20240503 14:14:46.965415 22867 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap starting.
I20240503 14:14:46.967761 22863 tablet_copy_client.cc:523] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:46.971217 22863 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: Bootstrap starting.
I20240503 14:14:46.975914 22867 log.cc:824] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:47.021780 22866 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.022565 22866 tablet_bootstrap.cc:492] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap complete.
I20240503 14:14:47.023185 22866 ts_tablet_manager.cc:1367] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: Time spent bootstrapping tablet: real 0.058s user 0.049s sys 0.002s
I20240503 14:14:47.030051 22866 raft_consensus.cc:348] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:47.030561 22866 raft_consensus.cc:729] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Initialized, Role: LEARNER
I20240503 14:14:47.031114 22866 consensus_queue.cc:260] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:47.031561 22867 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.032126 22867 tablet_bootstrap.cc:492] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap complete.
I20240503 14:14:47.032490 22867 ts_tablet_manager.cc:1367] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: Time spent bootstrapping tablet: real 0.067s user 0.051s sys 0.008s
I20240503 14:14:47.033035 22866 ts_tablet_manager.cc:1398] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240503 14:14:47.033638 22867 raft_consensus.cc:348] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:47.033985 22867 raft_consensus.cc:729] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Initialized, Role: LEARNER
I20240503 14:14:47.034135 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session c36ba962bcb34dbea55f4e55689bd38a-eeb40a038c4d404d9cb8ea693d9071d7 received from {username='slave'} at 127.10.145.9:39445
I20240503 14:14:47.034272 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session c36ba962bcb34dbea55f4e55689bd38a-eeb40a038c4d404d9cb8ea693d9071d7 on tablet eeb40a038c4d404d9cb8ea693d9071d7 with peer c36ba962bcb34dbea55f4e55689bd38a
I20240503 14:14:47.034266 22867 consensus_queue.cc:260] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:47.041885 22867 ts_tablet_manager.cc:1398] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240503 14:14:47.042797 20625 tablet_copy_service.cc:342] P 80260781ae364d049c2500ff76a06ce8: Request end of tablet copy session c36ba962bcb34dbea55f4e55689bd38a-91743575cf4a4f38b5b43265539237a3 received from {username='slave'} at 127.10.145.9:54867
I20240503 14:14:47.042989 20625 tablet_copy_service.cc:433] P 80260781ae364d049c2500ff76a06ce8: ending tablet copy session c36ba962bcb34dbea55f4e55689bd38a-91743575cf4a4f38b5b43265539237a3 on tablet 91743575cf4a4f38b5b43265539237a3 with peer c36ba962bcb34dbea55f4e55689bd38a
I20240503 14:14:47.054668 22863 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.055061 22863 tablet_bootstrap.cc:492] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: Bootstrap complete.
I20240503 14:14:47.055301 22863 ts_tablet_manager.cc:1367] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: Time spent bootstrapping tablet: real 0.084s user 0.041s sys 0.003s
I20240503 14:14:47.055954 22863 raft_consensus.cc:348] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:47.056206 22863 raft_consensus.cc:729] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Initialized, Role: LEARNER
I20240503 14:14:47.056421 22863 consensus_queue.cc:260] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:47.056991 22863 ts_tablet_manager.cc:1398] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:47.057984 18420 tablet_copy_service.cc:342] P ea366e6e15a24c52b1ea5a154f942568: Request end of tablet copy session 80260781ae364d049c2500ff76a06ce8-963ecc30d66b43a4ad149f596bd288d9 received from {username='slave'} at 127.10.145.8:41697
I20240503 14:14:47.058123 18420 tablet_copy_service.cc:433] P ea366e6e15a24c52b1ea5a154f942568: ending tablet copy session 80260781ae364d049c2500ff76a06ce8-963ecc30d66b43a4ad149f596bd288d9 on tablet 963ecc30d66b43a4ad149f596bd288d9 with peer 80260781ae364d049c2500ff76a06ce8
I20240503 14:14:47.059496 22893 tablet_copy_client.cc:523] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.063040 22893 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap starting.
I20240503 14:14:47.063266 22892 tablet_copy_client.cc:523] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.066812 22892 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: Bootstrap starting.
I20240503 14:14:47.107896 22892 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.108347 22892 tablet_bootstrap.cc:492] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: Bootstrap complete.
I20240503 14:14:47.108707 22892 ts_tablet_manager.cc:1367] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: Time spent bootstrapping tablet: real 0.042s user 0.040s sys 0.002s
I20240503 14:14:47.109406 22892 raft_consensus.cc:348] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:47.109777 22892 raft_consensus.cc:729] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a632094ba4554a4895d6c0b538b81af8, State: Initialized, Role: LEARNER
I20240503 14:14:47.110037 22892 consensus_queue.cc:260] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:47.110822 22892 ts_tablet_manager.cc:1398] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:47.111656 18679 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session a632094ba4554a4895d6c0b538b81af8-79373bd79a934eaab67bbcf2208e7b5b received from {username='slave'} at 127.10.145.7:47505
I20240503 14:14:47.111824 18679 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session a632094ba4554a4895d6c0b538b81af8-79373bd79a934eaab67bbcf2208e7b5b on tablet 79373bd79a934eaab67bbcf2208e7b5b with peer a632094ba4554a4895d6c0b538b81af8
I20240503 14:14:47.114523 22893 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.115005 22893 tablet_bootstrap.cc:492] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: Bootstrap complete.
I20240503 14:14:47.115367 22893 ts_tablet_manager.cc:1367] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent bootstrapping tablet: real 0.052s user 0.046s sys 0.005s
I20240503 14:14:47.116329 22893 raft_consensus.cc:348] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:47.116559 22893 raft_consensus.cc:729] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2e6a3f22c7ae444e96f6d35cca602e62, State: Initialized, Role: LEARNER
I20240503 14:14:47.116762 22893 consensus_queue.cc:260] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: NON_VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: true } }
I20240503 14:14:47.118309 22893 ts_tablet_manager.cc:1398] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:47.118970 18550 tablet_copy_service.cc:342] P 55339e38e7aa4bb7b7b42aa206108722: Request end of tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-19b1b6be3bda47829f53765282eae9f9 received from {username='slave'} at 127.10.145.5:59583
I20240503 14:14:47.119125 18550 tablet_copy_service.cc:433] P 55339e38e7aa4bb7b7b42aa206108722: ending tablet copy session 2e6a3f22c7ae444e96f6d35cca602e62-19b1b6be3bda47829f53765282eae9f9 on tablet 19b1b6be3bda47829f53765282eae9f9 with peer 2e6a3f22c7ae444e96f6d35cca602e62
I20240503 14:14:47.134686 22928 tablet_copy_client.cc:523] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.136946 22919 tablet_copy_client.cc:523] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.137948 22928 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap starting.
I20240503 14:14:47.140455 22919 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: Bootstrap starting.
I20240503 14:14:47.175490 22955 tablet_copy_client.cc:523] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.180784 22955 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: Bootstrap starting.
I20240503 14:14:47.188531 22928 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.188987 22928 tablet_bootstrap.cc:492] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap complete.
I20240503 14:14:47.189311 22928 ts_tablet_manager.cc:1367] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent bootstrapping tablet: real 0.051s user 0.033s sys 0.014s
I20240503 14:14:47.190189 22928 raft_consensus.cc:348] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:47.190476 22928 raft_consensus.cc:729] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Initialized, Role: LEARNER
I20240503 14:14:47.190716 22928 consensus_queue.cc:260] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:47.191310 22928 ts_tablet_manager.cc:1398] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:47.193418 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-eedc3cfb2dff4f9084446d89b9b3c0f4 received from {username='slave'} at 127.10.145.6:41609
I20240503 14:14:47.193562 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-eedc3cfb2dff4f9084446d89b9b3c0f4 on tablet eedc3cfb2dff4f9084446d89b9b3c0f4 with peer 69cbbbedfe03437b8c95b7bb730dbe66
I20240503 14:14:47.208774 22919 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.209472 22919 tablet_bootstrap.cc:492] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: Bootstrap complete.
I20240503 14:14:47.209822 22919 ts_tablet_manager.cc:1367] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: Time spent bootstrapping tablet: real 0.069s user 0.041s sys 0.010s
I20240503 14:14:47.210687 22919 raft_consensus.cc:348] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:47.210997 22919 raft_consensus.cc:729] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a632094ba4554a4895d6c0b538b81af8, State: Initialized, Role: LEARNER
I20240503 14:14:47.211236 22919 consensus_queue.cc:260] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:47.211910 22919 ts_tablet_manager.cc:1398] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:47.212914 20625 tablet_copy_service.cc:342] P 80260781ae364d049c2500ff76a06ce8: Request end of tablet copy session a632094ba4554a4895d6c0b538b81af8-ea3ea2e3d87f4c099abbf5193c239a8b received from {username='slave'} at 127.10.145.7:48145
I20240503 14:14:47.213052 20625 tablet_copy_service.cc:433] P 80260781ae364d049c2500ff76a06ce8: ending tablet copy session a632094ba4554a4895d6c0b538b81af8-ea3ea2e3d87f4c099abbf5193c239a8b on tablet ea3ea2e3d87f4c099abbf5193c239a8b with peer a632094ba4554a4895d6c0b538b81af8
I20240503 14:14:47.222162 22985 tablet_copy_client.cc:523] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.224205 22955 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.224684 22955 tablet_bootstrap.cc:492] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: Bootstrap complete.
I20240503 14:14:47.224972 22955 ts_tablet_manager.cc:1367] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: Time spent bootstrapping tablet: real 0.044s user 0.028s sys 0.009s
I20240503 14:14:47.225890 22955 raft_consensus.cc:348] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:47.226157 22955 raft_consensus.cc:729] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80260781ae364d049c2500ff76a06ce8, State: Initialized, Role: LEARNER
I20240503 14:14:47.226390 22955 consensus_queue.cc:260] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: true } }
I20240503 14:14:47.226873 22985 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap starting.
I20240503 14:14:47.227173 22955 ts_tablet_manager.cc:1398] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:14:47.228040 18550 tablet_copy_service.cc:342] P 55339e38e7aa4bb7b7b42aa206108722: Request end of tablet copy session 80260781ae364d049c2500ff76a06ce8-7c63b35eef494f0e8928c4ca0f74ec32 received from {username='slave'} at 127.10.145.8:40531
I20240503 14:14:47.228201 18550 tablet_copy_service.cc:433] P 55339e38e7aa4bb7b7b42aa206108722: ending tablet copy session 80260781ae364d049c2500ff76a06ce8-7c63b35eef494f0e8928c4ca0f74ec32 on tablet 7c63b35eef494f0e8928c4ca0f74ec32 with peer 80260781ae364d049c2500ff76a06ce8
I20240503 14:14:47.254140 23000 tablet_copy_client.cc:523] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.257998 23000 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap starting.
I20240503 14:14:47.258291 20475 raft_consensus.cc:1203] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:14:47.281647 22985 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.285717 22985 tablet_bootstrap.cc:492] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: Bootstrap complete.
I20240503 14:14:47.286060 22985 ts_tablet_manager.cc:1367] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent bootstrapping tablet: real 0.059s user 0.045s sys 0.004s
I20240503 14:14:47.286895 22985 raft_consensus.cc:348] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:47.287197 22985 raft_consensus.cc:729] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 69cbbbedfe03437b8c95b7bb730dbe66, State: Initialized, Role: LEARNER
I20240503 14:14:47.287418 22985 consensus_queue.cc:260] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: true } }
I20240503 14:14:47.288154 22985 ts_tablet_manager.cc:1398] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:47.289036 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-4cb29bdb71f24b46b32e7b97d0172c20 received from {username='slave'} at 127.10.145.6:41609
I20240503 14:14:47.289161 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session 69cbbbedfe03437b8c95b7bb730dbe66-4cb29bdb71f24b46b32e7b97d0172c20 on tablet 4cb29bdb71f24b46b32e7b97d0172c20 with peer 69cbbbedfe03437b8c95b7bb730dbe66
I20240503 14:14:47.296427 23015 tablet_copy_client.cc:523] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.299787 23015 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap starting.
I20240503 14:14:47.303687 21578 raft_consensus.cc:1203] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:14:47.306929 23000 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.307459 23000 tablet_bootstrap.cc:492] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap complete.
I20240503 14:14:47.307794 23000 ts_tablet_manager.cc:1367] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: Time spent bootstrapping tablet: real 0.050s user 0.042s sys 0.002s
I20240503 14:14:47.308706 23000 raft_consensus.cc:348] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:47.309023 23000 raft_consensus.cc:729] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Initialized, Role: LEARNER
I20240503 14:14:47.309257 23000 consensus_queue.cc:260] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:47.310084 23000 ts_tablet_manager.cc:1398] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:47.311075 18550 tablet_copy_service.cc:342] P 55339e38e7aa4bb7b7b42aa206108722: Request end of tablet copy session c36ba962bcb34dbea55f4e55689bd38a-7f9705f607a94835ad7e6c20668b80ad received from {username='slave'} at 127.10.145.9:56393
I20240503 14:14:47.311250 18550 tablet_copy_service.cc:433] P 55339e38e7aa4bb7b7b42aa206108722: ending tablet copy session c36ba962bcb34dbea55f4e55689bd38a-7f9705f607a94835ad7e6c20668b80ad on tablet 7f9705f607a94835ad7e6c20668b80ad with peer c36ba962bcb34dbea55f4e55689bd38a
I20240503 14:14:47.316545 20341 raft_consensus.cc:1203] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.327544 20604 raft_consensus.cc:1203] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.349895 23015 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.350402 23015 tablet_bootstrap.cc:492] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap complete.
I20240503 14:14:47.350713 23015 ts_tablet_manager.cc:1367] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: Time spent bootstrapping tablet: real 0.051s user 0.043s sys 0.000s
I20240503 14:14:47.351502 23015 raft_consensus.cc:348] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:47.351786 23015 raft_consensus.cc:729] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Initialized, Role: LEARNER
I20240503 14:14:47.351996 23015 consensus_queue.cc:260] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:47.352821 23015 ts_tablet_manager.cc:1398] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:47.354017 18420 tablet_copy_service.cc:342] P ea366e6e15a24c52b1ea5a154f942568: Request end of tablet copy session c36ba962bcb34dbea55f4e55689bd38a-8c337ef0405e4856bd2a608a6a602a49 received from {username='slave'} at 127.10.145.9:39915
I20240503 14:14:47.354182 18420 tablet_copy_service.cc:433] P ea366e6e15a24c52b1ea5a154f942568: ending tablet copy session c36ba962bcb34dbea55f4e55689bd38a-8c337ef0405e4856bd2a608a6a602a49 on tablet 8c337ef0405e4856bd2a608a6a602a49 with peer c36ba962bcb34dbea55f4e55689bd38a
I20240503 14:14:47.384754 21578 raft_consensus.cc:1203] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.388671 23022 tablet_copy_client.cc:523] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.391623 23022 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap starting.
I20240503 14:14:47.424726 23022 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.425201 23022 tablet_bootstrap.cc:492] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap complete.
I20240503 14:14:47.425518 23022 ts_tablet_manager.cc:1367] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.009s
I20240503 14:14:47.426294 23022 raft_consensus.cc:348] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:47.426555 23022 raft_consensus.cc:729] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Initialized, Role: LEARNER
I20240503 14:14:47.426741 23022 consensus_queue.cc:260] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:47.427397 23022 ts_tablet_manager.cc:1398] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:47.428239 18679 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session c36ba962bcb34dbea55f4e55689bd38a-7570d5bce0e5481d8dded653617be5a7 received from {username='slave'} at 127.10.145.9:51127
I20240503 14:14:47.428419 18679 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session c36ba962bcb34dbea55f4e55689bd38a-7570d5bce0e5481d8dded653617be5a7 on tablet 7570d5bce0e5481d8dded653617be5a7 with peer c36ba962bcb34dbea55f4e55689bd38a
I20240503 14:14:47.440687 20475 raft_consensus.cc:1203] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:14:47.523669 23044 tablet_copy_client.cc:523] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.526883 23044 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: Bootstrap starting.
I20240503 14:14:47.546258 20604 raft_consensus.cc:1203] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.556888 23050 tablet_copy_client.cc:523] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:47.560164 23050 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: Bootstrap starting.
I20240503 14:14:47.560926 23044 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.561504 23044 tablet_bootstrap.cc:492] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: Bootstrap complete.
I20240503 14:14:47.561787 23044 ts_tablet_manager.cc:1367] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: Time spent bootstrapping tablet: real 0.035s user 0.031s sys 0.005s
I20240503 14:14:47.562633 23044 raft_consensus.cc:348] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:47.562899 23044 raft_consensus.cc:729] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a632094ba4554a4895d6c0b538b81af8, State: Initialized, Role: LEARNER
I20240503 14:14:47.563095 23044 consensus_queue.cc:260] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:47.563594 23044 ts_tablet_manager.cc:1398] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:47.564275 18679 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session a632094ba4554a4895d6c0b538b81af8-9577e6d3801f42f89afe8a48a77031dd received from {username='slave'} at 127.10.145.7:47505
I20240503 14:14:47.564438 18679 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session a632094ba4554a4895d6c0b538b81af8-9577e6d3801f42f89afe8a48a77031dd on tablet 9577e6d3801f42f89afe8a48a77031dd with peer a632094ba4554a4895d6c0b538b81af8
I20240503 14:14:47.574333 20341 raft_consensus.cc:1203] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.590256 20210 raft_consensus.cc:1203] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.599337 23050 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:47.599823 23050 tablet_bootstrap.cc:492] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: Bootstrap complete.
I20240503 14:14:47.600138 23050 ts_tablet_manager.cc:1367] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: Time spent bootstrapping tablet: real 0.040s user 0.032s sys 0.007s
I20240503 14:14:47.600862 20475 raft_consensus.cc:1203] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.600981 23050 raft_consensus.cc:348] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:47.601246 23050 raft_consensus.cc:729] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a632094ba4554a4895d6c0b538b81af8, State: Initialized, Role: LEARNER
I20240503 14:14:47.601493 23050 consensus_queue.cc:260] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: NON_VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: true } }
I20240503 14:14:47.602151 23050 ts_tablet_manager.cc:1398] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:47.604174 20362 tablet_copy_service.cc:342] P 69cbbbedfe03437b8c95b7bb730dbe66: Request end of tablet copy session a632094ba4554a4895d6c0b538b81af8-f0c1a7f5739f4fa59271254e501fb095 received from {username='slave'} at 127.10.145.7:48885
I20240503 14:14:47.604388 20362 tablet_copy_service.cc:433] P 69cbbbedfe03437b8c95b7bb730dbe66: ending tablet copy session a632094ba4554a4895d6c0b538b81af8-f0c1a7f5739f4fa59271254e501fb095 on tablet f0c1a7f5739f4fa59271254e501fb095 with peer a632094ba4554a4895d6c0b538b81af8
I20240503 14:14:47.622946 20475 raft_consensus.cc:1203] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:14:47.667477 21578 raft_consensus.cc:1203] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.688035 21578 raft_consensus.cc:1203] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.719166 21578 raft_consensus.cc:1203] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.723282 23027 raft_consensus.cc:1050] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: attempting to promote NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 to VOTER
I20240503 14:14:47.724071 23027 consensus_queue.cc:237] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:47.726897 20341 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.728312 23361 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:47.728507 20210 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.729110 23361 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:47.736131 20342 raft_consensus.cc:1203] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:47.738972 23361 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:47.740561 18657 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.741259 23436 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:47.741912 20342 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:47.745870 18128 catalog_manager.cc:5484] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:47.745024 20210 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:47.749035 18657 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:47.811973 23450 raft_consensus.cc:1050] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568: attempting to promote NON_VOTER 80260781ae364d049c2500ff76a06ce8 to VOTER
I20240503 14:14:47.812902 23450 consensus_queue.cc:237] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:47.815591 20342 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.815989 18657 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.816044 20604 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.816239 23449 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:47.816579 23450 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:47.816963 23452 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:47.819970 23450 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:47.820504 20604 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:47.820606 18657 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:47.824345 20342 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:47.824860 18119 catalog_manager.cc:5484] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 7 to 8, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:47.835253 18399 consensus_queue.cc:237] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:47.837028 20342 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.837066 20604 raft_consensus.cc:1261] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:47.837658 23450 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.838265 23450 consensus_queue.cc:1035] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.840471 23450 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:47.841495 20604 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:47.842793 20342 raft_consensus.cc:2932] T 963ecc30d66b43a4ad149f596bd288d9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:47.844309 18114 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 963ecc30d66b43a4ad149f596bd288d9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:47.846791 18128 catalog_manager.cc:5484] T 963ecc30d66b43a4ad149f596bd288d9 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:47.852010 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 963ecc30d66b43a4ad149f596bd288d9 with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:47.853253 23458 tablet_replica.cc:330] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:47.853858 23458 raft_consensus.cc:2229] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:47.854322 23458 raft_consensus.cc:2258] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:47.856071 23457 raft_consensus.cc:1050] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8: attempting to promote NON_VOTER a632094ba4554a4895d6c0b538b81af8 to VOTER
I20240503 14:14:47.856889 23458 ts_tablet_manager.cc:1875] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:47.857158 23457 consensus_queue.cc:237] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:47.859882 18529 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:47.860828 23456 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.861709 20475 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:47.862766 18657 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:47.863736 23458 ts_tablet_manager.cc:1888] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:47.863852 23458 log.cc:1196] T 963ecc30d66b43a4ad149f596bd288d9 P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/963ecc30d66b43a4ad149f596bd288d9
I20240503 14:14:47.864267 23021 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.864692 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet 963ecc30d66b43a4ad149f596bd288d9 (table rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]) successfully deleted
I20240503 14:14:47.865029 23456 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.868788 23460 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:47.869524 18529 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:47.872951 18657 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:47.873319 20475 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:47.873998 18120 catalog_manager.cc:5484] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 reported cstate change: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "80260781ae364d049c2500ff76a06ce8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:47.878844 20604 consensus_queue.cc:237] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:47.881220 18660 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:47.881799 23021 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.881897 20475 raft_consensus.cc:1261] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:47.882393 23460 consensus_queue.cc:1035] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.884272 23021 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:47.885063 18657 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:47.887151 20475 raft_consensus.cc:2932] T ea3ea2e3d87f4c099abbf5193c239a8b P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:47.890241 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea3ea2e3d87f4c099abbf5193c239a8b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:47.891340 18121 catalog_manager.cc:5484] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 reported cstate change: config changed from index 9 to 10, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 2 leader_uuid: "80260781ae364d049c2500ff76a06ce8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:47.895691 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3ea2e3d87f4c099abbf5193c239a8b with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:47.896445 23464 tablet_replica.cc:330] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:47.896771 23464 raft_consensus.cc:2229] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:47.897068 23464 raft_consensus.cc:2258] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:47.897925 23464 ts_tablet_manager.cc:1875] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:47.902570 23464 ts_tablet_manager.cc:1888] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:14:47.902674 23464 log.cc:1196] T ea3ea2e3d87f4c099abbf5193c239a8b P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/ea3ea2e3d87f4c099abbf5193c239a8b
I20240503 14:14:47.903369 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet ea3ea2e3d87f4c099abbf5193c239a8b (table rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]) successfully deleted
I20240503 14:14:47.918956 23021 raft_consensus.cc:1050] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8: attempting to promote NON_VOTER c36ba962bcb34dbea55f4e55689bd38a to VOTER
I20240503 14:14:47.919890 23021 consensus_queue.cc:237] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:47.931444 21578 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a [term 2 LEARNER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:47.932188 23021 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.932883 18399 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:47.933415 18657 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:47.934335 23021 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.936443 23027 raft_consensus.cc:1050] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722: attempting to promote NON_VOTER 80260781ae364d049c2500ff76a06ce8 to VOTER
I20240503 14:14:47.937244 23027 consensus_queue.cc:237] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:47.938416 23456 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:47.951213 23456 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:47.953680 23465 raft_consensus.cc:1050] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER c36ba962bcb34dbea55f4e55689bd38a to VOTER
I20240503 14:14:47.954756 23465 consensus_queue.cc:237] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:47.957135 20475 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.957342 20604 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8 [term 1 LEARNER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.957765 23436 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:47.958314 18657 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.958168 18660 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:47.958854 23436 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:47.959357 23361 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:47.961158 18398 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:47.961125 21578 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:47.964403 18128 catalog_manager.cc:5484] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 8 to 9, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "80260781ae364d049c2500ff76a06ce8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:47.971688 20605 consensus_queue.cc:237] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:47.974767 18398 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:47.975486 23021 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.979578 21577 raft_consensus.cc:1261] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Refusing update from remote peer 80260781ae364d049c2500ff76a06ce8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:47.983301 23361 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:47.986567 20604 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:47.987112 23021 consensus_queue.cc:1035] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:47.991824 23457 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:47.993412 21577 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.994122 23342 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:47.994683 20210 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:47.996775 23374 raft_consensus.cc:1050] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER a632094ba4554a4895d6c0b538b81af8 to VOTER
I20240503 14:14:47.998154 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 91743575cf4a4f38b5b43265539237a3 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:47.999226 21578 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:47.999866 23374 consensus_queue.cc:237] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:48.000862 20475 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:48.003584 18121 catalog_manager.cc:5484] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8 reported cstate change: config changed from index 7 to 8, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:48.007665 23342 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.007954 18119 catalog_manager.cc:5484] T 91743575cf4a4f38b5b43265539237a3 P c36ba962bcb34dbea55f4e55689bd38a reported cstate change: config changed from index 9 to 10, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 2 leader_uuid: "80260781ae364d049c2500ff76a06ce8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.013849 18660 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80260781ae364d049c2500ff76a06ce8 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:48.017789 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 91743575cf4a4f38b5b43265539237a3 with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:48.018189 23458 tablet_replica.cc:330] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:48.018414 23458 raft_consensus.cc:2229] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:48.019271 23361 raft_consensus.cc:1050] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722: attempting to promote NON_VOTER 2e6a3f22c7ae444e96f6d35cca602e62 to VOTER
I20240503 14:14:48.019377 23458 raft_consensus.cc:2258] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:48.020151 23361 consensus_queue.cc:237] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:48.020450 23458 ts_tablet_manager.cc:1875] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:48.026067 18529 consensus_queue.cc:237] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:48.026459 20475 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:48.027379 20210 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:48.028540 20475 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:48.028820 20604 raft_consensus.cc:1261] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:48.029223 23436 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.029803 23436 consensus_queue.cc:1035] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.030802 21671 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.031494 21671 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.032333 18399 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.032311 23436 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:48.037459 20474 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:48.040144 23465 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.041419 18399 raft_consensus.cc:2932] T 91743575cf4a4f38b5b43265539237a3 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.042526 23458 ts_tablet_manager.cc:1888] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:14:48.042742 23458 log.cc:1196] T 91743575cf4a4f38b5b43265539237a3 P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/91743575cf4a4f38b5b43265539237a3
I20240503 14:14:48.042483 20604 raft_consensus.cc:2932] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:48.043538 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet 91743575cf4a4f38b5b43265539237a3 (table rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]) successfully deleted
I20240503 14:14:48.046388 18400 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:48.047055 21671 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.047349 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c63b35eef494f0e8928c4ca0f74ec32 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:48.051657 18120 catalog_manager.cc:5484] T 7c63b35eef494f0e8928c4ca0f74ec32 P 80260781ae364d049c2500ff76a06ce8 reported cstate change: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } }
I20240503 14:14:48.056545 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c63b35eef494f0e8928c4ca0f74ec32 with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:48.056980 23458 tablet_replica.cc:330] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:48.057209 23458 raft_consensus.cc:2229] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:48.057648 23458 raft_consensus.cc:2258] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:48.058840 23458 ts_tablet_manager.cc:1875] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:48.065372 18657 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.065730 20342 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.066349 23436 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.067011 23374 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.067279 23474 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.070708 18399 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.070214 20475 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.075178 23469 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.076054 21577 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.079128 20210 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 LEARNER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.081041 18128 catalog_manager.cc:5484] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.082614 20209 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.084089 23436 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.088141 22788 raft_consensus.cc:1050] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: attempting to promote NON_VOTER c36ba962bcb34dbea55f4e55689bd38a to VOTER
I20240503 14:14:48.088932 18119 catalog_manager.cc:5484] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 reported cstate change: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.089102 22788 consensus_queue.cc:237] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:48.090994 18399 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.081408 20210 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.101758 23474 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:48.107236 18660 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:48.107141 20342 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:48.107817 18121 catalog_manager.cc:5484] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 7 to 8, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.067950 23458 ts_tablet_manager.cc:1888] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:48.114012 23458 log.cc:1196] T 7c63b35eef494f0e8928c4ca0f74ec32 P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/7c63b35eef494f0e8928c4ca0f74ec32
I20240503 14:14:48.114898 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet 7c63b35eef494f0e8928c4ca0f74ec32 (table rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]) successfully deleted
I20240503 14:14:48.115442 23493 raft_consensus.cc:1050] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66: attempting to promote NON_VOTER a632094ba4554a4895d6c0b538b81af8 to VOTER
I20240503 14:14:48.116464 23493 consensus_queue.cc:237] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:48.132364 18529 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:48.132946 18657 consensus_queue.cc:237] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:48.133093 23301 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.134845 20475 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:48.135349 20210 raft_consensus.cc:1261] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:48.135352 23486 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:48.135947 23486 consensus_queue.cc:1035] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:48.138273 23374 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.139055 20210 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.141829 20474 raft_consensus.cc:2932] T 79373bd79a934eaab67bbcf2208e7b5b P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.142454 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 79373bd79a934eaab67bbcf2208e7b5b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:48.144942 18119 catalog_manager.cc:5484] T 79373bd79a934eaab67bbcf2208e7b5b P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 9 to 10, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 2 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:48.149464 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet 79373bd79a934eaab67bbcf2208e7b5b with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:48.154012 23486 raft_consensus.cc:1050] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER c36ba962bcb34dbea55f4e55689bd38a to VOTER
I20240503 14:14:48.155236 23486 consensus_queue.cc:237] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:48.156858 23502 tablet_replica.cc:330] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:48.157120 23502 raft_consensus.cc:2229] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:48.157600 23502 raft_consensus.cc:2258] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:48.158413 20474 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8 [term 2 LEARNER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:48.158669 23502 ts_tablet_manager.cc:1875] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:48.159241 23301 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.164964 20604 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.165688 23452 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.173256 23502 ts_tablet_manager.cc:1888] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:14:48.173529 23502 log.cc:1196] T 79373bd79a934eaab67bbcf2208e7b5b P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/79373bd79a934eaab67bbcf2208e7b5b
I20240503 14:14:48.174466 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet 79373bd79a934eaab67bbcf2208e7b5b (table rebalance_test_table_6 [id=6b0f739ac0ee4c98a3035bf4ee8d6dbe]) successfully deleted
I20240503 14:14:48.178107 23301 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.179034 18529 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.180145 21577 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.181824 23452 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.184024 20474 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.184629 18119 catalog_manager.cc:5484] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 8 to 9, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.195636 18269 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.196354 22788 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.197930 23469 raft_consensus.cc:1050] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 to VOTER
I20240503 14:14:48.198675 20342 consensus_queue.cc:237] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:48.198769 23469 consensus_queue.cc:237] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:48.201251 18269 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:48.203960 23342 raft_consensus.cc:1050] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER 69cbbbedfe03437b8c95b7bb730dbe66 to VOTER
I20240503 14:14:48.204838 23342 consensus_queue.cc:237] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:48.211601 18529 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:48.212695 23493 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:48.215555 23493 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.216284 20475 raft_consensus.cc:1261] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Refusing update from remote peer 69cbbbedfe03437b8c95b7bb730dbe66: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:48.217554 23301 consensus_queue.cc:1035] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:48.218015 20605 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.219072 18528 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.216797 23452 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.222565 18529 consensus_queue.cc:237] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } }
I20240503 14:14:48.224684 20210 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:48.224700 20342 raft_consensus.cc:1261] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:48.225253 23436 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.225049 20475 raft_consensus.cc:2932] T f0c1a7f5739f4fa59271254e501fb095 P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.226816 23474 consensus_queue.cc:1035] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.227149 21671 raft_consensus.cc:1050] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER a632094ba4554a4895d6c0b538b81af8 to VOTER
I20240503 14:14:48.227977 21671 consensus_queue.cc:237] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:48.228955 18128 catalog_manager.cc:5484] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 9 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 2 leader_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.227526 20208 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:48.230947 23436 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:48.232000 18119 catalog_manager.cc:5484] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 reported cstate change: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.234169 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c1a7f5739f4fa59271254e501fb095 with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:48.235139 20341 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:48.241135 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 19b1b6be3bda47829f53765282eae9f9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:48.243243 18119 catalog_manager.cc:5484] T 19b1b6be3bda47829f53765282eae9f9 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:48.240530 20208 raft_consensus.cc:2932] T 19b1b6be3bda47829f53765282eae9f9 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } }
I20240503 14:14:48.249450 20475 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [term 1 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.249462 20210 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.249794 20209 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.250123 23374 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.250326 23342 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.250757 23374 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.252739 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b1b6be3bda47829f53765282eae9f9 with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:48.253111 23458 tablet_replica.cc:330] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:48.253362 23458 raft_consensus.cc:2229] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:48.253763 23458 raft_consensus.cc:2258] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:48.256139 18113 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f0c1a7f5739f4fa59271254e501fb095 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:48.256739 23458 ts_tablet_manager.cc:1875] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:48.257206 20342 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.258802 20342 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.259469 23342 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.260687 23342 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.261058 18398 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.257961 21577 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.265143 23458 ts_tablet_manager.cc:1888] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:48.265317 23458 log.cc:1196] T 19b1b6be3bda47829f53765282eae9f9 P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/19b1b6be3bda47829f53765282eae9f9
I20240503 14:14:48.266048 23342 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.266191 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet 19b1b6be3bda47829f53765282eae9f9 (table rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]) successfully deleted
I20240503 14:14:48.268375 20604 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.268990 23469 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.269110 18528 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.269932 23469 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.274602 21578 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.276458 18530 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.276774 20605 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.276464 18529 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.278309 23374 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.279028 23374 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.284729 23512 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.287374 23374 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240503 14:14:48.265663 18268 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.304220 23465 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.307022 23511 tablet_replica.cc:330] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:48.307385 23511 raft_consensus.cc:2229] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:48.307991 23511 raft_consensus.cc:2258] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:48.308920 23511 ts_tablet_manager.cc:1875] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:48.309372 18530 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.313899 20342 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.315313 20605 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.316078 18128 catalog_manager.cc:5484] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.326025 23516 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.326660 18529 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.330948 20604 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.333181 18121 catalog_manager.cc:5484] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.339154 23027 raft_consensus.cc:1050] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722: attempting to promote NON_VOTER c36ba962bcb34dbea55f4e55689bd38a to VOTER
I20240503 14:14:48.340027 23027 consensus_queue.cc:237] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:48.341969 23342 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.342370 23486 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.348795 18657 consensus_queue.cc:237] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:48.349792 21578 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.350643 18529 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.351351 20604 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:48.347906 20475 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.351960 23516 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.352442 18121 catalog_manager.cc:5484] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 7 to 8, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:48.354534 23374 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.358346 18120 catalog_manager.cc:5484] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:48.359318 20604 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.359295 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7570d5bce0e5481d8dded653617be5a7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:48.361579 18398 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.365047 18268 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.365146 21576 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.365787 23474 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.366027 20474 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:48.367570 23361 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.370178 18120 catalog_manager.cc:5484] T 7570d5bce0e5481d8dded653617be5a7 P 80260781ae364d049c2500ff76a06ce8 reported cstate change: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.373523 23027 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.376123 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet 7570d5bce0e5481d8dded653617be5a7 with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:48.376525 23464 tablet_replica.cc:330] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:48.376724 23464 raft_consensus.cc:2229] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:48.376964 23464 raft_consensus.cc:2258] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:48.377903 23464 ts_tablet_manager.cc:1875] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:48.383277 20209 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.389505 18268 consensus_queue.cc:237] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:48.391849 20209 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:48.391950 20341 raft_consensus.cc:1261] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:48.391902 23464 ts_tablet_manager.cc:1888] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:48.392136 23464 log.cc:1196] T 7570d5bce0e5481d8dded653617be5a7 P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/7570d5bce0e5481d8dded653617be5a7
I20240503 14:14:48.392697 23467 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.392514 23474 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.393261 23467 consensus_queue.cc:1035] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:48.393358 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet 7570d5bce0e5481d8dded653617be5a7 (table rebalance_test_table_1 [id=32f5c2950e2b4f50914479099197fc97]) successfully deleted
I20240503 14:14:48.395264 20340 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.395778 21577 raft_consensus.cc:1261] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:48.396566 23516 consensus_queue.cc:1035] T 7570d5bce0e5481d8dded653617be5a7 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.398236 21577 raft_consensus.cc:2932] T 7570d5bce0e5481d8dded653617be5a7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.398216 23467 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.398347 20475 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.398646 18267 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.403450 21578 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.404937 20209 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.405699 18125 catalog_manager.cc:5484] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.407423 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eedc3cfb2dff4f9084446d89b9b3c0f4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:48.410969 18119 catalog_manager.cc:5484] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 2e6a3f22c7ae444e96f6d35cca602e62 reported cstate change: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.416363 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet eedc3cfb2dff4f9084446d89b9b3c0f4 with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:48.416826 23502 tablet_replica.cc:330] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:48.417054 23502 raft_consensus.cc:2229] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:48.417098 18529 consensus_queue.cc:237] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:48.417325 23502 raft_consensus.cc:2258] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:48.417948 20340 raft_consensus.cc:2932] T eedc3cfb2dff4f9084446d89b9b3c0f4 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:48.418385 23502 ts_tablet_manager.cc:1875] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:48.422046 20475 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:48.422488 21578 raft_consensus.cc:1261] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:48.423130 23534 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.425525 23474 consensus_queue.cc:1035] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:48.426038 23436 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.426671 23502 ts_tablet_manager.cc:1888] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:48.426784 23502 log.cc:1196] T eedc3cfb2dff4f9084446d89b9b3c0f4 P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/eedc3cfb2dff4f9084446d89b9b3c0f4
I20240503 14:14:48.427476 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet eedc3cfb2dff4f9084446d89b9b3c0f4 (table rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]) successfully deleted
I20240503 14:14:48.429409 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7f9705f607a94835ad7e6c20668b80ad with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:48.429062 20209 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a632094ba4554a4895d6c0b538b81af8 (127.10.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:48.431062 20475 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.434144 18120 catalog_manager.cc:5484] T 7f9705f607a94835ad7e6c20668b80ad P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 8 to 9, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.440806 23511 ts_tablet_manager.cc:1888] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240503 14:14:48.440850 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f9705f607a94835ad7e6c20668b80ad with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:48.441424 23511 log.cc:1196] T f0c1a7f5739f4fa59271254e501fb095 P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/f0c1a7f5739f4fa59271254e501fb095
I20240503 14:14:48.443320 21577 raft_consensus.cc:2932] T 7f9705f607a94835ad7e6c20668b80ad P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:48.444453 23511 tablet_replica.cc:330] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:48.444463 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet f0c1a7f5739f4fa59271254e501fb095 (table rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]) successfully deleted
I20240503 14:14:48.444655 23511 raft_consensus.cc:2229] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:48.444908 23511 raft_consensus.cc:2258] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:48.449131 23511 ts_tablet_manager.cc:1875] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:48.454531 23511 ts_tablet_manager.cc:1888] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:48.454710 23511 log.cc:1196] T 7f9705f607a94835ad7e6c20668b80ad P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/7f9705f607a94835ad7e6c20668b80ad
I20240503 14:14:48.455516 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet 7f9705f607a94835ad7e6c20668b80ad (table rebalance_test_table_3 [id=a4091d9ac30e41efb7300f841de51eeb]) successfully deleted
I20240503 14:14:48.565165 18269 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eeb40a038c4d404d9cb8ea693d9071d7"
dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38846
I20240503 14:14:48.565416 18269 raft_consensus.cc:595] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:48.582561 18398 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c337ef0405e4856bd2a608a6a602a49"
dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34354
I20240503 14:14:48.582868 18398 raft_consensus.cc:595] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:48.600479 18529 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "71a326db29114366951a67b9aa84ac03"
dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53908
I20240503 14:14:48.600802 18529 raft_consensus.cc:595] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:48.616688 23342 raft_consensus.cc:979] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: : Instructing follower ea366e6e15a24c52b1ea5a154f942568 to start an election
I20240503 14:14:48.616911 23342 raft_consensus.cc:1067] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Signalling peer ea366e6e15a24c52b1ea5a154f942568 to start an election
I20240503 14:14:48.617709 18398 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eeb40a038c4d404d9cb8ea693d9071d7"
dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
from {username='slave'} at 127.10.145.1:51293
I20240503 14:14:48.618266 18398 raft_consensus.cc:482] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:48.618391 18398 raft_consensus.cc:3037] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.619493 18657 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9577e6d3801f42f89afe8a48a77031dd"
dest_uuid: "1c72210d694344d4b4604140088a6471"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47982
I20240503 14:14:48.619772 18657 raft_consensus.cc:595] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:48.621618 18398 raft_consensus.cc:504] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:48.622551 18398 leader_election.cc:288] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 2 election: Requested vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605), c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329)
I20240503 14:14:48.622974 20210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb40a038c4d404d9cb8ea693d9071d7" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2e6a3f22c7ae444e96f6d35cca602e62"
I20240503 14:14:48.623276 20210 raft_consensus.cc:3037] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.623286 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb40a038c4d404d9cb8ea693d9071d7" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:48.623476 18269 raft_consensus.cc:3032] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:48.623407 21577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb40a038c4d404d9cb8ea693d9071d7" candidate_uuid: "ea366e6e15a24c52b1ea5a154f942568" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c36ba962bcb34dbea55f4e55689bd38a"
I20240503 14:14:48.623564 18269 raft_consensus.cc:729] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:48.623731 21577 raft_consensus.cc:3037] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.623842 18269 consensus_queue.cc:260] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:48.624293 18269 raft_consensus.cc:3037] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.626171 20210 raft_consensus.cc:2454] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 2.
I20240503 14:14:48.626916 18269 raft_consensus.cc:2454] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 2.
I20240503 14:14:48.626940 21577 raft_consensus.cc:2454] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea366e6e15a24c52b1ea5a154f942568 in term 2.
I20240503 14:14:48.628298 18336 leader_election.cc:302] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2e6a3f22c7ae444e96f6d35cca602e62, c36ba962bcb34dbea55f4e55689bd38a, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:48.628634 23452 raft_consensus.cc:2781] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:48.628800 23452 raft_consensus.cc:686] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 2 LEADER]: Becoming Leader. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Running, Role: LEADER
I20240503 14:14:48.629149 23452 consensus_queue.cc:237] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:48.630326 23361 raft_consensus.cc:979] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: : Instructing follower 1c72210d694344d4b4604140088a6471 to start an election
I20240503 14:14:48.630517 23361 raft_consensus.cc:1067] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Signalling peer 1c72210d694344d4b4604140088a6471 to start an election
I20240503 14:14:48.631474 18657 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71a326db29114366951a67b9aa84ac03"
dest_uuid: "1c72210d694344d4b4604140088a6471"
from {username='slave'} at 127.10.145.3:46539
I20240503 14:14:48.631708 18657 raft_consensus.cc:482] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:48.631841 18657 raft_consensus.cc:3037] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.632947 18120 catalog_manager.cc:5484] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: term changed from 1 to 2, leader changed from 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) to ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2). New cstate: current_term: 2 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.635504 18657 raft_consensus.cc:504] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:48.636667 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a326db29114366951a67b9aa84ac03" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
I20240503 14:14:48.636886 18529 raft_consensus.cc:3032] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:48.637004 18529 raft_consensus.cc:729] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Running, Role: LEADER
I20240503 14:14:48.637495 18657 leader_election.cc:288] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 2 election: Requested vote from peers 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857), 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605), 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735)
I20240503 14:14:48.637373 18529 consensus_queue.cc:260] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:48.637526 20342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a326db29114366951a67b9aa84ac03" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "69cbbbedfe03437b8c95b7bb730dbe66"
I20240503 14:14:48.637868 20342 raft_consensus.cc:3037] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.637889 18529 raft_consensus.cc:3037] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.639814 20342 raft_consensus.cc:2454] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 2.
I20240503 14:14:48.639900 18529 raft_consensus.cc:2454] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 2.
I20240503 14:14:48.640585 18596 leader_election.cc:302] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c72210d694344d4b4604140088a6471, 55339e38e7aa4bb7b7b42aa206108722, 69cbbbedfe03437b8c95b7bb730dbe66; no voters:
I20240503 14:14:48.640718 20210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a326db29114366951a67b9aa84ac03" candidate_uuid: "1c72210d694344d4b4604140088a6471" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2e6a3f22c7ae444e96f6d35cca602e62"
I20240503 14:14:48.640841 23486 raft_consensus.cc:2781] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:48.640919 20210 raft_consensus.cc:3037] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.640996 23486 raft_consensus.cc:686] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 2 LEADER]: Becoming Leader. State: Replica: 1c72210d694344d4b4604140088a6471, State: Running, Role: LEADER
I20240503 14:14:48.641269 23486 consensus_queue.cc:237] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:48.643616 20210 raft_consensus.cc:2454] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c72210d694344d4b4604140088a6471 in term 2.
I20240503 14:14:48.644814 18120 catalog_manager.cc:5484] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 reported cstate change: term changed from 1 to 2, leader changed from 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) to 1c72210d694344d4b4604140088a6471 (127.10.145.4). New cstate: current_term: 2 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.659327 18269 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4cb29bdb71f24b46b32e7b97d0172c20"
dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38864
I20240503 14:14:48.659521 18269 raft_consensus.cc:595] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:48.679220 23452 raft_consensus.cc:979] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: : Instructing follower c36ba962bcb34dbea55f4e55689bd38a to start an election
I20240503 14:14:48.679481 22788 raft_consensus.cc:1067] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Signalling peer c36ba962bcb34dbea55f4e55689bd38a to start an election
I20240503 14:14:48.680233 21577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c337ef0405e4856bd2a608a6a602a49"
dest_uuid: "c36ba962bcb34dbea55f4e55689bd38a"
from {username='slave'} at 127.10.145.2:33359
I20240503 14:14:48.680522 21577 raft_consensus.cc:482] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:48.680646 21577 raft_consensus.cc:3037] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.682336 18269 consensus_queue.cc:237] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:48.682889 21577 raft_consensus.cc:504] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:48.685050 20210 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:48.685240 18529 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:48.685853 23342 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:48.686234 21577 leader_election.cc:288] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 election: Requested vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107)
I20240503 14:14:48.686429 23342 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:48.686554 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c337ef0405e4856bd2a608a6a602a49" candidate_uuid: "c36ba962bcb34dbea55f4e55689bd38a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ea366e6e15a24c52b1ea5a154f942568"
I20240503 14:14:48.686769 18398 raft_consensus.cc:3032] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:48.686863 18398 raft_consensus.cc:729] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ea366e6e15a24c52b1ea5a154f942568, State: Running, Role: LEADER
I20240503 14:14:48.687108 18398 consensus_queue.cc:260] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
W20240503 14:14:48.687530 18205 consensus_peers.cc:485] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 -> Peer c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329): Couldn't send request to peer c36ba962bcb34dbea55f4e55689bd38a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2c5c6f4317649fab75d01e339b76403. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:48.687544 18398 raft_consensus.cc:3037] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.687543 18269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c337ef0405e4856bd2a608a6a602a49" candidate_uuid: "c36ba962bcb34dbea55f4e55689bd38a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:48.687770 18269 raft_consensus.cc:3037] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.690408 18398 raft_consensus.cc:2454] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c36ba962bcb34dbea55f4e55689bd38a in term 2.
I20240503 14:14:48.690512 18269 raft_consensus.cc:2454] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c36ba962bcb34dbea55f4e55689bd38a in term 2.
I20240503 14:14:48.690977 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c337ef0405e4856bd2a608a6a602a49" candidate_uuid: "c36ba962bcb34dbea55f4e55689bd38a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "80260781ae364d049c2500ff76a06ce8"
I20240503 14:14:48.691056 21391 leader_election.cc:302] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7ceaaad4bc544c4487859a6671e66838, c36ba962bcb34dbea55f4e55689bd38a, ea366e6e15a24c52b1ea5a154f942568; no voters:
I20240503 14:14:48.691236 20604 raft_consensus.cc:3037] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.691500 23271 raft_consensus.cc:2781] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:48.691987 23271 raft_consensus.cc:686] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 2 LEADER]: Becoming Leader. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Running, Role: LEADER
I20240503 14:14:48.692549 23271 consensus_queue.cc:237] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:48.694012 20604 raft_consensus.cc:2454] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c36ba962bcb34dbea55f4e55689bd38a in term 2.
I20240503 14:14:48.696712 23342 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:48.697049 18120 catalog_manager.cc:5484] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a reported cstate change: term changed from 1 to 2, leader changed from ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) to c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9). New cstate: current_term: 2 leader_uuid: "c36ba962bcb34dbea55f4e55689bd38a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:48.700888 20210 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:48.704166 18528 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:48.705384 18120 catalog_manager.cc:5484] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.713726 23342 raft_consensus.cc:979] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: : Instructing follower 55339e38e7aa4bb7b7b42aa206108722 to start an election
I20240503 14:14:48.713912 23342 raft_consensus.cc:1067] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Signalling peer 55339e38e7aa4bb7b7b42aa206108722 to start an election
I20240503 14:14:48.714637 18529 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cb29bdb71f24b46b32e7b97d0172c20"
dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
from {username='slave'} at 127.10.145.1:58531
I20240503 14:14:48.714901 18529 raft_consensus.cc:482] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:48.715003 18529 raft_consensus.cc:3037] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.717835 18529 raft_consensus.cc:504] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:48.718873 18529 leader_election.cc:288] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 2 election: Requested vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 80260781ae364d049c2500ff76a06ce8 (127.10.145.8:44107), 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735)
I20240503 14:14:48.719463 18267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29bdb71f24b46b32e7b97d0172c20" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7ceaaad4bc544c4487859a6671e66838"
I20240503 14:14:48.719504 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29bdb71f24b46b32e7b97d0172c20" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "80260781ae364d049c2500ff76a06ce8"
I20240503 14:14:48.719570 20342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29bdb71f24b46b32e7b97d0172c20" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "69cbbbedfe03437b8c95b7bb730dbe66"
I20240503 14:14:48.719698 18267 raft_consensus.cc:3032] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:48.719710 20604 raft_consensus.cc:3037] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.719775 18267 raft_consensus.cc:729] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ceaaad4bc544c4487859a6671e66838, State: Running, Role: LEADER
I20240503 14:14:48.719787 20342 raft_consensus.cc:3037] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.720063 18267 consensus_queue.cc:260] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:48.720494 18267 raft_consensus.cc:3037] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.721697 20604 raft_consensus.cc:2454] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 2.
I20240503 14:14:48.722489 20342 raft_consensus.cc:2454] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 2.
I20240503 14:14:48.723055 18267 raft_consensus.cc:2454] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 2.
I20240503 14:14:48.723435 18463 leader_election.cc:302] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 55339e38e7aa4bb7b7b42aa206108722, 69cbbbedfe03437b8c95b7bb730dbe66, 7ceaaad4bc544c4487859a6671e66838; no voters:
I20240503 14:14:48.723989 23534 raft_consensus.cc:2781] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:48.724174 23534 raft_consensus.cc:686] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 LEADER]: Becoming Leader. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Running, Role: LEADER
I20240503 14:14:48.724504 23534 consensus_queue.cc:237] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:48.727963 18128 catalog_manager.cc:5484] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: term changed from 1 to 2, leader changed from 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) to 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3). New cstate: current_term: 2 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.729480 18267 consensus_queue.cc:237] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:48.742291 18398 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:48.742487 18205 consensus_peers.cc:485] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 -> Peer c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329): Couldn't send request to peer c36ba962bcb34dbea55f4e55689bd38a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a39fbf1d8f6f46ccab3c2b992ed0c63e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:48.742588 20342 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:48.742940 23465 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:48.743486 23342 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:48.746068 23469 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:48.746966 18398 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:48.747908 20342 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:48.753108 18121 catalog_manager.cc:5484] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.760754 23516 raft_consensus.cc:979] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: : Instructing follower 55339e38e7aa4bb7b7b42aa206108722 to start an election
I20240503 14:14:48.760950 23516 raft_consensus.cc:1067] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Signalling peer 55339e38e7aa4bb7b7b42aa206108722 to start an election
I20240503 14:14:48.763269 18529 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9577e6d3801f42f89afe8a48a77031dd"
dest_uuid: "55339e38e7aa4bb7b7b42aa206108722"
from {username='slave'} at 127.10.145.4:47049
I20240503 14:14:48.763516 18529 raft_consensus.cc:482] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:48.763661 18529 raft_consensus.cc:3037] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.766291 18529 raft_consensus.cc:504] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:48.767220 18529 leader_election.cc:288] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 2 election: Requested vote from peers 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411), 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605), a632094ba4554a4895d6c0b538b81af8 (127.10.145.7:38307)
I20240503 14:14:48.767964 20210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9577e6d3801f42f89afe8a48a77031dd" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2e6a3f22c7ae444e96f6d35cca602e62"
I20240503 14:14:48.768190 20210 raft_consensus.cc:3037] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.768218 18657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9577e6d3801f42f89afe8a48a77031dd" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1c72210d694344d4b4604140088a6471"
I20240503 14:14:48.768424 18657 raft_consensus.cc:3032] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:48.768342 20475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9577e6d3801f42f89afe8a48a77031dd" candidate_uuid: "55339e38e7aa4bb7b7b42aa206108722" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a632094ba4554a4895d6c0b538b81af8"
I20240503 14:14:48.768498 18657 raft_consensus.cc:729] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c72210d694344d4b4604140088a6471, State: Running, Role: LEADER
I20240503 14:14:48.768517 18529 consensus_queue.cc:237] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:48.768690 20475 raft_consensus.cc:3037] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.768812 18657 consensus_queue.cc:260] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:48.769274 18657 raft_consensus.cc:3037] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.771224 20210 raft_consensus.cc:2454] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 2.
I20240503 14:14:48.771785 20475 raft_consensus.cc:2454] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 2.
I20240503 14:14:48.772420 18657 raft_consensus.cc:2454] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55339e38e7aa4bb7b7b42aa206108722 in term 2.
I20240503 14:14:48.772460 18466 leader_election.cc:302] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2e6a3f22c7ae444e96f6d35cca602e62, 55339e38e7aa4bb7b7b42aa206108722, a632094ba4554a4895d6c0b538b81af8; no voters:
I20240503 14:14:48.773124 23534 raft_consensus.cc:2781] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:48.773319 23534 raft_consensus.cc:686] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 2 LEADER]: Becoming Leader. State: Replica: 55339e38e7aa4bb7b7b42aa206108722, State: Running, Role: LEADER
I20240503 14:14:48.773604 23534 consensus_queue.cc:237] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:48.778787 18128 catalog_manager.cc:5484] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: term changed from 1 to 2, leader changed from 1c72210d694344d4b4604140088a6471 (127.10.145.4) to 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3). New cstate: current_term: 2 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:48.780011 20604 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:48.780280 18267 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:48.780926 18466 consensus_peers.cc:485] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 -> Peer c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329): Couldn't send request to peer c36ba962bcb34dbea55f4e55689bd38a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c35376539bc944e59b74bfe59d5feb9e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:48.781170 23361 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:48.781569 23474 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:48.785004 23361 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:48.788832 18267 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:48.798053 18121 catalog_manager.cc:5484] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:48.788877 20604 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:49.066421 20210 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.067085 23518 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.072403 20210 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.072928 22788 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.075961 23699 ts_tablet_manager.cc:902] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:49.077212 23699 tablet_copy_client.cc:308] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:49.077941 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet f2c5c6f4317649fab75d01e339b76403 from peer c36ba962bcb34dbea55f4e55689bd38a ({username='slave'} at 127.10.145.9:39445)
I20240503 14:14:49.078109 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet f2c5c6f4317649fab75d01e339b76403 from peer c36ba962bcb34dbea55f4e55689bd38a at {username='slave'} at 127.10.145.9:39445: session id = c36ba962bcb34dbea55f4e55689bd38a-f2c5c6f4317649fab75d01e339b76403
I20240503 14:14:49.080052 18289 tablet_copy_source_session.cc:205] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:49.081264 23699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c5c6f4317649fab75d01e339b76403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:49.086078 23699 tablet_copy_client.cc:783] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:49.086328 23699 tablet_copy_client.cc:647] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:49.087743 23703 tablet_copy_client.cc:1002] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:49.111924 18267 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.112603 22788 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.119735 21577 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.120643 23714 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.169468 18267 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.170042 23534 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.182301 20210 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.182993 23534 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.194789 23741 ts_tablet_manager.cc:902] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: Initiating tablet copy from peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857)
I20240503 14:14:49.197669 20475 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.198591 23534 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.201628 23741 tablet_copy_client.cc:308] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:38857
I20240503 14:14:49.202410 18550 tablet_copy_service.cc:140] P 55339e38e7aa4bb7b7b42aa206108722: Received BeginTabletCopySession request for tablet c35376539bc944e59b74bfe59d5feb9e from peer c36ba962bcb34dbea55f4e55689bd38a ({username='slave'} at 127.10.145.9:56393)
I20240503 14:14:49.202598 18550 tablet_copy_service.cc:161] P 55339e38e7aa4bb7b7b42aa206108722: Beginning new tablet copy session on tablet c35376539bc944e59b74bfe59d5feb9e from peer c36ba962bcb34dbea55f4e55689bd38a at {username='slave'} at 127.10.145.9:56393: session id = c36ba962bcb34dbea55f4e55689bd38a-c35376539bc944e59b74bfe59d5feb9e
I20240503 14:14:49.205029 18550 tablet_copy_source_session.cc:205] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:49.205751 18529 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.206589 23518 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.208431 18657 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.209358 23027 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.210335 23741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35376539bc944e59b74bfe59d5feb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:49.220818 23741 tablet_copy_client.cc:783] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:49.221097 23741 tablet_copy_client.cc:647] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:49.223246 23745 tablet_copy_client.cc:1002] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:49.224570 18528 consensus_queue.cc:237] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:49.226838 20475 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:49.227384 20210 raft_consensus.cc:1261] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:49.227456 23534 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:49.229830 23436 consensus_queue.cc:1035] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:49.229858 23027 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 55339e38e7aa4bb7b7b42aa206108722 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:49.231578 20210 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:49.234488 20475 raft_consensus.cc:2932] T 9577e6d3801f42f89afe8a48a77031dd P a632094ba4554a4895d6c0b538b81af8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:49.238796 18267 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer c36ba962bcb34dbea55f4e55689bd38a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.239763 23271 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.240936 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9577e6d3801f42f89afe8a48a77031dd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:49.246166 18121 catalog_manager.cc:5484] T 9577e6d3801f42f89afe8a48a77031dd P 2e6a3f22c7ae444e96f6d35cca602e62 reported cstate change: config changed from index 8 to 10, VOTER 1c72210d694344d4b4604140088a6471 (127.10.145.4) evicted. New cstate: current_term: 2 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } } }
I20240503 14:14:49.253932 20342 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.254703 23486 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.255303 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 9577e6d3801f42f89afe8a48a77031dd with delete_type TABLET_DATA_TOMBSTONED (TS 1c72210d694344d4b4604140088a6471 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39696
I20240503 14:14:49.259546 23755 tablet_replica.cc:330] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: stopping tablet replica
I20240503 14:14:49.260383 23755 raft_consensus.cc:2229] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:49.260594 23755 raft_consensus.cc:2258] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:49.262076 23755 ts_tablet_manager.cc:1875] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:49.269558 23755 ts_tablet_manager.cc:1888] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:14:49.269819 23755 log.cc:1196] T 9577e6d3801f42f89afe8a48a77031dd P 1c72210d694344d4b4604140088a6471: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/9577e6d3801f42f89afe8a48a77031dd
I20240503 14:14:49.271783 18112 catalog_manager.cc:4830] TS 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): tablet 9577e6d3801f42f89afe8a48a77031dd (table rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]) successfully deleted
I20240503 14:14:49.272485 20604 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Refusing update from remote peer c36ba962bcb34dbea55f4e55689bd38a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.274284 20340 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.274487 23271 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.274880 23027 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.279345 20605 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.280474 23474 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.298468 18398 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Refusing update from remote peer c36ba962bcb34dbea55f4e55689bd38a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:14:49.302733 23767 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:49.330564 21577 consensus_queue.cc:237] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:49.333418 18267 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Refusing update from remote peer c36ba962bcb34dbea55f4e55689bd38a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:49.333964 20604 raft_consensus.cc:1261] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Refusing update from remote peer c36ba962bcb34dbea55f4e55689bd38a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:49.334362 23765 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:49.336247 23765 consensus_queue.cc:1035] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:49.337217 23767 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:49.337939 18267 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:49.343360 20604 raft_consensus.cc:2932] T 8c337ef0405e4856bd2a608a6a602a49 P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:49.344084 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c337ef0405e4856bd2a608a6a602a49 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:49.345551 18121 catalog_manager.cc:5484] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a reported cstate change: config changed from index 8 to 10, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 2 leader_uuid: "c36ba962bcb34dbea55f4e55689bd38a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:49.351369 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c337ef0405e4856bd2a608a6a602a49 with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:49.355890 23778 ts_tablet_manager.cc:902] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:49.371730 23778 tablet_copy_client.cc:308] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:49.371845 23779 tablet_replica.cc:330] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:49.372082 23779 raft_consensus.cc:2229] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:49.372501 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e from peer c36ba962bcb34dbea55f4e55689bd38a ({username='slave'} at 127.10.145.9:39445)
I20240503 14:14:49.372695 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet a39fbf1d8f6f46ccab3c2b992ed0c63e from peer c36ba962bcb34dbea55f4e55689bd38a at {username='slave'} at 127.10.145.9:39445: session id = c36ba962bcb34dbea55f4e55689bd38a-a39fbf1d8f6f46ccab3c2b992ed0c63e
I20240503 14:14:49.373006 23779 raft_consensus.cc:2258] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:49.374524 23779 ts_tablet_manager.cc:1875] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:49.375485 18289 tablet_copy_source_session.cc:205] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:49.377449 23778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:49.382397 23779 ts_tablet_manager.cc:1888] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:14:49.382635 23779 log.cc:1196] T 8c337ef0405e4856bd2a608a6a602a49 P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/8c337ef0405e4856bd2a608a6a602a49
I20240503 14:14:49.383920 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet 8c337ef0405e4856bd2a608a6a602a49 (table rebalance_test_table_0 [id=69a3ceacb5da449096a476bce97fab6c]) successfully deleted
I20240503 14:14:49.384479 23778 tablet_copy_client.cc:783] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:49.384718 23778 tablet_copy_client.cc:647] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:49.386483 23784 tablet_copy_client.cc:1002] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:49.457964 18657 consensus_queue.cc:237] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:49.460291 20342 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:49.460314 18398 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:49.460511 18596 consensus_peers.cc:485] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 -> Peer c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329): Couldn't send request to peer c36ba962bcb34dbea55f4e55689bd38a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bc67ebb58a246d9a8c47f0838a4831a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:49.460978 23516 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:49.461297 23518 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:49.464196 23486 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:49.465015 18398 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:49.467554 20342 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:49.471937 18121 catalog_manager.cc:5484] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 reported cstate change: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:14:49.638161 18657 consensus_queue.cc:237] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:49.640228 20342 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:49.640228 20210 raft_consensus.cc:1261] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:49.640828 23518 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:49.641165 23516 consensus_queue.cc:1035] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:49.643604 23486 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 1c72210d694344d4b4604140088a6471 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:49.644332 20342 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:49.646303 20210 raft_consensus.cc:2932] T 71a326db29114366951a67b9aa84ac03 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:49.647513 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71a326db29114366951a67b9aa84ac03 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:49.649363 18121 catalog_manager.cc:5484] T 71a326db29114366951a67b9aa84ac03 P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 8 to 10, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 2 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:49.653345 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a326db29114366951a67b9aa84ac03 with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:49.657804 23860 tablet_replica.cc:330] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:49.658025 23860 raft_consensus.cc:2229] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:49.658254 23860 raft_consensus.cc:2258] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:49.659102 23860 ts_tablet_manager.cc:1875] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:49.664080 23860 ts_tablet_manager.cc:1888] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:14:49.664204 23860 log.cc:1196] T 71a326db29114366951a67b9aa84ac03 P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/71a326db29114366951a67b9aa84ac03
I20240503 14:14:49.664955 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet 71a326db29114366951a67b9aa84ac03 (table rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]) successfully deleted
I20240503 14:14:49.713694 18398 consensus_queue.cc:237] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:49.715677 21577 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:49.715797 20210 raft_consensus.cc:1261] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Refusing update from remote peer ea366e6e15a24c52b1ea5a154f942568: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:49.716352 23862 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:49.716697 23863 consensus_queue.cc:1035] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:49.719082 23862 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P ea366e6e15a24c52b1ea5a154f942568 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:49.719938 21577 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:49.720402 20210 raft_consensus.cc:2932] T eeb40a038c4d404d9cb8ea693d9071d7 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:49.723601 18114 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eeb40a038c4d404d9cb8ea693d9071d7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:49.724611 18121 catalog_manager.cc:5484] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a reported cstate change: config changed from index 8 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 2 leader_uuid: "ea366e6e15a24c52b1ea5a154f942568" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:49.728365 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb40a038c4d404d9cb8ea693d9071d7 with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:49.729219 23864 tablet_replica.cc:330] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:49.729550 23864 raft_consensus.cc:2229] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:49.729769 23864 raft_consensus.cc:2258] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:49.730906 23864 ts_tablet_manager.cc:1875] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:49.735662 23864 ts_tablet_manager.cc:1888] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:14:49.735775 23864 log.cc:1196] T eeb40a038c4d404d9cb8ea693d9071d7 P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/eeb40a038c4d404d9cb8ea693d9071d7
I20240503 14:14:49.736498 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet eeb40a038c4d404d9cb8ea693d9071d7 (table rebalance_test_table_4 [id=fb406539c7904e128f596746c62fde25]) successfully deleted
I20240503 14:14:49.749358 18528 consensus_queue.cc:237] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } }
I20240503 14:14:49.751693 20342 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:49.751936 20604 raft_consensus.cc:1261] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:14:49.752367 23474 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:49.752740 23027 consensus_queue.cc:1035] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:14:49.755012 23027 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 55339e38e7aa4bb7b7b42aa206108722 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:49.755721 20342 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:49.757603 20604 raft_consensus.cc:2932] T 4cb29bdb71f24b46b32e7b97d0172c20 P 80260781ae364d049c2500ff76a06ce8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:49.759032 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4cb29bdb71f24b46b32e7b97d0172c20 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:49.762605 18120 catalog_manager.cc:5484] T 4cb29bdb71f24b46b32e7b97d0172c20 P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 8 to 10, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 2 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } }
I20240503 14:14:49.766594 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb29bdb71f24b46b32e7b97d0172c20 with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:49.767063 23864 tablet_replica.cc:330] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:49.767454 23864 raft_consensus.cc:2229] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:49.767930 23864 raft_consensus.cc:2258] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:49.768908 23864 ts_tablet_manager.cc:1875] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:49.774621 23864 ts_tablet_manager.cc:1888] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:14:49.774763 23864 log.cc:1196] T 4cb29bdb71f24b46b32e7b97d0172c20 P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/4cb29bdb71f24b46b32e7b97d0172c20
I20240503 14:14:49.775444 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet 4cb29bdb71f24b46b32e7b97d0172c20 (table rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]) successfully deleted
I20240503 14:14:49.864229 18267 consensus_queue.cc:237] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:49.866647 18528 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:14:49.866647 20210 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:14:49.866699 18205 consensus_peers.cc:485] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 -> Peer c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9:45329): Couldn't send request to peer c36ba962bcb34dbea55f4e55689bd38a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe3d20ad69474bcbae19bdd06cbfeb52. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:49.867228 23469 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:49.867547 23859 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:49.870098 23467 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:49.870790 18528 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:49.870829 20210 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:49.875521 18121 catalog_manager.cc:5484] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 6 to 7, NON_VOTER c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } } }
I20240503 14:14:50.068218 23933 ts_tablet_manager.cc:902] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: Initiating tablet copy from peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411)
I20240503 14:14:50.069384 23933 tablet_copy_client.cc:308] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:36411
I20240503 14:14:50.070083 18679 tablet_copy_service.cc:140] P 1c72210d694344d4b4604140088a6471: Received BeginTabletCopySession request for tablet 5bc67ebb58a246d9a8c47f0838a4831a from peer c36ba962bcb34dbea55f4e55689bd38a ({username='slave'} at 127.10.145.9:51127)
I20240503 14:14:50.070274 18679 tablet_copy_service.cc:161] P 1c72210d694344d4b4604140088a6471: Beginning new tablet copy session on tablet 5bc67ebb58a246d9a8c47f0838a4831a from peer c36ba962bcb34dbea55f4e55689bd38a at {username='slave'} at 127.10.145.9:51127: session id = c36ba962bcb34dbea55f4e55689bd38a-5bc67ebb58a246d9a8c47f0838a4831a
I20240503 14:14:50.072229 18679 tablet_copy_source_session.cc:205] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:50.073504 23933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc67ebb58a246d9a8c47f0838a4831a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:50.078176 23933 tablet_copy_client.cc:783] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:50.078437 23933 tablet_copy_client.cc:647] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:50.080029 23937 tablet_copy_client.cc:1002] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:50.334599 23963 ts_tablet_manager.cc:902] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: Initiating tablet copy from peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655)
I20240503 14:14:50.336403 23963 tablet_copy_client.cc:308] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:33655
I20240503 14:14:50.337889 18289 tablet_copy_service.cc:140] P 7ceaaad4bc544c4487859a6671e66838: Received BeginTabletCopySession request for tablet fe3d20ad69474bcbae19bdd06cbfeb52 from peer c36ba962bcb34dbea55f4e55689bd38a ({username='slave'} at 127.10.145.9:39445)
I20240503 14:14:50.338649 18289 tablet_copy_service.cc:161] P 7ceaaad4bc544c4487859a6671e66838: Beginning new tablet copy session on tablet fe3d20ad69474bcbae19bdd06cbfeb52 from peer c36ba962bcb34dbea55f4e55689bd38a at {username='slave'} at 127.10.145.9:39445: session id = c36ba962bcb34dbea55f4e55689bd38a-fe3d20ad69474bcbae19bdd06cbfeb52
I20240503 14:14:50.341598 18289 tablet_copy_source_session.cc:205] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:50.342945 23963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3d20ad69474bcbae19bdd06cbfeb52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:50.348289 23963 tablet_copy_client.cc:783] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:50.348635 23963 tablet_copy_client.cc:647] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:50.350407 23967 tablet_copy_client.cc:1002] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Injecting latency into file download: 1500
I20240503 14:14:50.588213 23699 tablet_copy_client.cc:523] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:50.591463 23699 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap starting.
I20240503 14:14:50.625981 23699 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:50.626515 23699 tablet_bootstrap.cc:492] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap complete.
I20240503 14:14:50.626894 23699 ts_tablet_manager.cc:1367] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.012s
I20240503 14:14:50.627677 23699 raft_consensus.cc:348] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:50.627885 23699 raft_consensus.cc:729] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Initialized, Role: LEARNER
I20240503 14:14:50.628085 23699 consensus_queue.cc:260] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:50.628578 23699 ts_tablet_manager.cc:1398] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:50.629258 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session c36ba962bcb34dbea55f4e55689bd38a-f2c5c6f4317649fab75d01e339b76403 received from {username='slave'} at 127.10.145.9:39445
I20240503 14:14:50.629460 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session c36ba962bcb34dbea55f4e55689bd38a-f2c5c6f4317649fab75d01e339b76403 on tablet f2c5c6f4317649fab75d01e339b76403 with peer c36ba962bcb34dbea55f4e55689bd38a
I20240503 14:14:50.723649 23741 tablet_copy_client.cc:523] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:50.727336 23741 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap starting.
I20240503 14:14:50.760623 23741 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:50.761145 23741 tablet_bootstrap.cc:492] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap complete.
I20240503 14:14:50.761447 23741 ts_tablet_manager.cc:1367] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: Time spent bootstrapping tablet: real 0.034s user 0.036s sys 0.000s
I20240503 14:14:50.762101 23741 raft_consensus.cc:348] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:50.762324 23741 raft_consensus.cc:729] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Initialized, Role: LEARNER
I20240503 14:14:50.762509 23741 consensus_queue.cc:260] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:50.763027 23741 ts_tablet_manager.cc:1398] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:14:50.763870 18550 tablet_copy_service.cc:342] P 55339e38e7aa4bb7b7b42aa206108722: Request end of tablet copy session c36ba962bcb34dbea55f4e55689bd38a-c35376539bc944e59b74bfe59d5feb9e received from {username='slave'} at 127.10.145.9:56393
I20240503 14:14:50.763993 18550 tablet_copy_service.cc:433] P 55339e38e7aa4bb7b7b42aa206108722: ending tablet copy session c36ba962bcb34dbea55f4e55689bd38a-c35376539bc944e59b74bfe59d5feb9e on tablet c35376539bc944e59b74bfe59d5feb9e with peer c36ba962bcb34dbea55f4e55689bd38a
I20240503 14:14:50.886869 23778 tablet_copy_client.cc:523] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:50.889851 23778 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap starting.
I20240503 14:14:50.933235 23778 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:50.933712 23778 tablet_bootstrap.cc:492] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap complete.
I20240503 14:14:50.934026 23778 ts_tablet_manager.cc:1367] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: Time spent bootstrapping tablet: real 0.044s user 0.042s sys 0.000s
I20240503 14:14:50.934860 23778 raft_consensus.cc:348] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:50.935142 21577 raft_consensus.cc:1203] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:50.935168 23778 raft_consensus.cc:729] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Initialized, Role: LEARNER
I20240503 14:14:50.935743 23778 consensus_queue.cc:260] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:50.936795 23778 ts_tablet_manager.cc:1398] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:50.939042 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session c36ba962bcb34dbea55f4e55689bd38a-a39fbf1d8f6f46ccab3c2b992ed0c63e received from {username='slave'} at 127.10.145.9:39445
I20240503 14:14:50.939214 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session c36ba962bcb34dbea55f4e55689bd38a-a39fbf1d8f6f46ccab3c2b992ed0c63e on tablet a39fbf1d8f6f46ccab3c2b992ed0c63e with peer c36ba962bcb34dbea55f4e55689bd38a
I20240503 14:14:50.986078 21577 raft_consensus.cc:1203] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:51.094631 21577 raft_consensus.cc:1203] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:51.532081 24022 raft_consensus.cc:1050] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER c36ba962bcb34dbea55f4e55689bd38a to VOTER
I20240503 14:14:51.532898 24022 consensus_queue.cc:237] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:51.535873 21577 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:51.535868 20210 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:51.535914 18528 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:51.536588 24022 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:51.536999 24101 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:51.537359 23859 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:51.541217 23859 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.541991 21577 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.545670 20210 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.545907 18528 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.546733 18128 catalog_manager.cc:5484] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:51.555563 18267 consensus_queue.cc:237] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:51.557629 20210 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:51.557672 21577 raft_consensus.cc:1261] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:51.558346 24023 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:51.558729 24101 consensus_queue.cc:1035] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:51.561317 24023 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.562057 21577 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.562654 20210 raft_consensus.cc:2932] T f2c5c6f4317649fab75d01e339b76403 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.564450 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2c5c6f4317649fab75d01e339b76403 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:51.567535 18121 catalog_manager.cc:5484] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a reported cstate change: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.572561 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c5c6f4317649fab75d01e339b76403 with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:51.573385 24112 tablet_replica.cc:330] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:51.573594 24112 raft_consensus.cc:2229] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:51.573832 24112 raft_consensus.cc:2258] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:51.574877 24112 ts_tablet_manager.cc:1875] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:51.577630 23027 raft_consensus.cc:1050] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722: attempting to promote NON_VOTER c36ba962bcb34dbea55f4e55689bd38a to VOTER
I20240503 14:14:51.578480 23027 consensus_queue.cc:237] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:51.579834 24112 ts_tablet_manager.cc:1888] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:51.579967 24112 log.cc:1196] T f2c5c6f4317649fab75d01e339b76403 P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/f2c5c6f4317649fab75d01e339b76403
I20240503 14:14:51.580437 23933 tablet_copy_client.cc:523] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:51.580924 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet f2c5c6f4317649fab75d01e339b76403 (table rebalance_test_table_8 [id=ba734cc6886b4f46b983de26474deddb]) successfully deleted
I20240503 14:14:51.581173 20604 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:51.581936 18267 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:51.581879 24111 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:51.582595 24111 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:51.583365 21577 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:51.584225 23933 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap starting.
I20240503 14:14:51.584878 24111 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:51.588280 24110 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.588970 21577 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.594166 18267 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.596932 24101 raft_consensus.cc:1050] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER c36ba962bcb34dbea55f4e55689bd38a to VOTER
I20240503 14:14:51.596807 18121 catalog_manager.cc:5484] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:51.598233 24101 consensus_queue.cc:237] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:51.603706 20342 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:51.604094 18398 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:51.604766 23859 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:51.605106 24022 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:51.610229 20604 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } attrs { replace: true } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.611526 21577 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:51.612265 23859 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:51.615427 18528 consensus_queue.cc:237] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:51.618430 21576 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:51.619036 20605 raft_consensus.cc:1261] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Refusing update from remote peer 55339e38e7aa4bb7b7b42aa206108722: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:51.619926 24110 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:51.624444 24110 consensus_queue.cc:1035] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:51.628196 24114 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.628715 21577 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.633060 18115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c35376539bc944e59b74bfe59d5feb9e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:51.632951 20605 raft_consensus.cc:2932] T c35376539bc944e59b74bfe59d5feb9e P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.639183 18120 catalog_manager.cc:5484] T c35376539bc944e59b74bfe59d5feb9e P 55339e38e7aa4bb7b7b42aa206108722 reported cstate change: config changed from index 8 to 9, VOTER 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "55339e38e7aa4bb7b7b42aa206108722" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:51.650884 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet c35376539bc944e59b74bfe59d5feb9e with delete_type TABLET_DATA_TOMBSTONED (TS 7ceaaad4bc544c4487859a6671e66838 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60596
I20240503 14:14:51.655591 24123 tablet_replica.cc:330] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838: stopping tablet replica
I20240503 14:14:51.655963 24123 raft_consensus.cc:2229] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:51.657999 24123 raft_consensus.cc:2258] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:51.660054 24123 ts_tablet_manager.cc:1875] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:51.660492 23859 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.661070 18398 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.666365 18120 catalog_manager.cc:5484] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 reported cstate change: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.667477 20342 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.667970 24123 ts_tablet_manager.cc:1888] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:51.668098 24123 log.cc:1196] T c35376539bc944e59b74bfe59d5feb9e P 7ceaaad4bc544c4487859a6671e66838: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/c35376539bc944e59b74bfe59d5feb9e
I20240503 14:14:51.668998 18112 catalog_manager.cc:4830] TS 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): tablet c35376539bc944e59b74bfe59d5feb9e (table rebalance_test_table_7 [id=94922275ec674d218bca18df289695e4]) successfully deleted
I20240503 14:14:51.670884 21578 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.677244 18267 consensus_queue.cc:237] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:51.679829 21578 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:51.679929 20342 raft_consensus.cc:1261] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:51.680552 24022 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:51.680976 24023 consensus_queue.cc:1035] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:51.683887 24023 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.687623 20340 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.692159 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:51.692512 21578 raft_consensus.cc:2932] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.693735 18121 catalog_manager.cc:5484] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:51.698405 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet a39fbf1d8f6f46ccab3c2b992ed0c63e with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:51.702734 24127 tablet_replica.cc:330] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:51.702971 24127 raft_consensus.cc:2229] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:51.703212 24127 raft_consensus.cc:2258] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:51.704252 24127 ts_tablet_manager.cc:1875] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:51.708303 23933 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:51.708657 23933 tablet_bootstrap.cc:492] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap complete.
I20240503 14:14:51.708897 23933 ts_tablet_manager.cc:1367] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: Time spent bootstrapping tablet: real 0.125s user 0.055s sys 0.041s
I20240503 14:14:51.709687 23933 raft_consensus.cc:348] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:51.709923 23933 raft_consensus.cc:729] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Initialized, Role: LEARNER
I20240503 14:14:51.710068 23933 consensus_queue.cc:260] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:51.710595 23933 ts_tablet_manager.cc:1398] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:51.712263 18679 tablet_copy_service.cc:342] P 1c72210d694344d4b4604140088a6471: Request end of tablet copy session c36ba962bcb34dbea55f4e55689bd38a-5bc67ebb58a246d9a8c47f0838a4831a received from {username='slave'} at 127.10.145.9:51127
I20240503 14:14:51.712410 18679 tablet_copy_service.cc:433] P 1c72210d694344d4b4604140088a6471: ending tablet copy session c36ba962bcb34dbea55f4e55689bd38a-5bc67ebb58a246d9a8c47f0838a4831a on tablet 5bc67ebb58a246d9a8c47f0838a4831a with peer c36ba962bcb34dbea55f4e55689bd38a
I20240503 14:14:51.715461 24127 ts_tablet_manager.cc:1888] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:51.715626 24127 log.cc:1196] T a39fbf1d8f6f46ccab3c2b992ed0c63e P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/a39fbf1d8f6f46ccab3c2b992ed0c63e
I20240503 14:14:51.716426 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet a39fbf1d8f6f46ccab3c2b992ed0c63e (table rebalance_test_table_5 [id=b5b33d202c3f498a93af3c66bef86084]) successfully deleted
I20240503 14:14:51.850816 23963 tablet_copy_client.cc:523] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:51.853921 23963 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap starting.
I20240503 14:14:51.886540 23963 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:51.909641 23963 tablet_bootstrap.cc:492] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: Bootstrap complete.
I20240503 14:14:51.909999 23963 ts_tablet_manager.cc:1367] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: Time spent bootstrapping tablet: real 0.056s user 0.024s sys 0.009s
I20240503 14:14:51.910743 23963 raft_consensus.cc:348] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:51.910998 23963 raft_consensus.cc:729] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c36ba962bcb34dbea55f4e55689bd38a, State: Initialized, Role: LEARNER
I20240503 14:14:51.911201 23963 consensus_queue.cc:260] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: true } }
I20240503 14:14:51.911710 23963 ts_tablet_manager.cc:1398] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:51.912590 18289 tablet_copy_service.cc:342] P 7ceaaad4bc544c4487859a6671e66838: Request end of tablet copy session c36ba962bcb34dbea55f4e55689bd38a-fe3d20ad69474bcbae19bdd06cbfeb52 received from {username='slave'} at 127.10.145.9:39445
I20240503 14:14:51.912750 18289 tablet_copy_service.cc:433] P 7ceaaad4bc544c4487859a6671e66838: ending tablet copy session c36ba962bcb34dbea55f4e55689bd38a-fe3d20ad69474bcbae19bdd06cbfeb52 on tablet fe3d20ad69474bcbae19bdd06cbfeb52 with peer c36ba962bcb34dbea55f4e55689bd38a
I20240503 14:14:51.922160 21578 raft_consensus.cc:1203] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:51.932754 21578 raft_consensus.cc:1203] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:14:52.371346 24183 raft_consensus.cc:1050] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838: attempting to promote NON_VOTER c36ba962bcb34dbea55f4e55689bd38a to VOTER
I20240503 14:14:52.372006 24183 consensus_queue.cc:237] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:52.374559 21578 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:52.374565 20209 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:52.374648 18528 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:52.375166 24186 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:52.375566 24183 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:52.375847 24023 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:52.383217 24183 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.383811 20210 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.384701 21578 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.390872 18127 catalog_manager.cc:5484] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 reported cstate change: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.392027 18528 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "55339e38e7aa4bb7b7b42aa206108722" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 38857 } attrs { replace: true } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.398114 18267 consensus_queue.cc:237] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:52.400735 21578 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:52.400803 20210 raft_consensus.cc:1261] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Refusing update from remote peer 7ceaaad4bc544c4487859a6671e66838: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:52.401990 24183 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:52.402356 24023 consensus_queue.cc:1035] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:52.405668 24194 raft_consensus.cc:1050] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471: attempting to promote NON_VOTER c36ba962bcb34dbea55f4e55689bd38a to VOTER
I20240503 14:14:52.406502 24194 consensus_queue.cc:237] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:52.406244 24183 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.407440 21578 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.409103 18398 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:52.409782 23968 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:52.410704 21577 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a [term 1 LEARNER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:52.410744 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe3d20ad69474bcbae19bdd06cbfeb52 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:52.412358 20342 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:14:52.413087 23968 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:52.413743 23968 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:52.414736 18121 catalog_manager.cc:5484] T fe3d20ad69474bcbae19bdd06cbfeb52 P 7ceaaad4bc544c4487859a6671e66838 reported cstate change: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "7ceaaad4bc544c4487859a6671e66838" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:14:52.416807 23968 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.419031 20210 raft_consensus.cc:2932] T fe3d20ad69474bcbae19bdd06cbfeb52 P 2e6a3f22c7ae444e96f6d35cca602e62 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.419978 20342 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.420933 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3d20ad69474bcbae19bdd06cbfeb52 with delete_type TABLET_DATA_TOMBSTONED (TS 55339e38e7aa4bb7b7b42aa206108722 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38664
I20240503 14:14:52.421536 21577 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.421536 18398 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.426836 18120 catalog_manager.cc:5484] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 7 to 8, c36ba962bcb34dbea55f4e55689bd38a (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } attrs { replace: true } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.431928 24201 tablet_replica.cc:330] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722: stopping tablet replica
I20240503 14:14:52.432229 24201 raft_consensus.cc:2229] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:52.433398 24201 raft_consensus.cc:2258] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:52.434449 24201 ts_tablet_manager.cc:1875] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:52.437429 18657 consensus_queue.cc:237] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:52.440008 20342 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:52.440722 24195 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:52.441316 24201 ts_tablet_manager.cc:1888] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:52.441511 24201 log.cc:1196] T fe3d20ad69474bcbae19bdd06cbfeb52 P 55339e38e7aa4bb7b7b42aa206108722: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/fe3d20ad69474bcbae19bdd06cbfeb52
I20240503 14:14:52.442337 18115 catalog_manager.cc:4830] TS 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): tablet fe3d20ad69474bcbae19bdd06cbfeb52 (table rebalance_test_table_9 [id=25b24c7006964d62a36c1ee1e769d7a3]) successfully deleted
I20240503 14:14:52.442402 21577 raft_consensus.cc:1261] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Refusing update from remote peer 1c72210d694344d4b4604140088a6471: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:14:52.443002 24195 consensus_queue.cc:1035] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:14:52.443600 24197 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 1c72210d694344d4b4604140088a6471 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.446880 18112 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5bc67ebb58a246d9a8c47f0838a4831a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:52.447098 21578 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.448206 20342 raft_consensus.cc:2932] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.454125 18128 catalog_manager.cc:5484] T 5bc67ebb58a246d9a8c47f0838a4831a P 69cbbbedfe03437b8c95b7bb730dbe66 reported cstate change: config changed from index 8 to 9, VOTER ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "1c72210d694344d4b4604140088a6471" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } } }
I20240503 14:14:52.458967 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc67ebb58a246d9a8c47f0838a4831a with delete_type TABLET_DATA_TOMBSTONED (TS ea366e6e15a24c52b1ea5a154f942568 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52196
I20240503 14:14:52.460098 24207 tablet_replica.cc:330] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568: stopping tablet replica
I20240503 14:14:52.460387 24207 raft_consensus.cc:2229] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:52.460796 24207 raft_consensus.cc:2258] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:52.462319 24207 ts_tablet_manager.cc:1875] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:52.467572 24207 ts_tablet_manager.cc:1888] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:14:52.467705 24207 log.cc:1196] T 5bc67ebb58a246d9a8c47f0838a4831a P ea366e6e15a24c52b1ea5a154f942568: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/5bc67ebb58a246d9a8c47f0838a4831a
I20240503 14:14:52.468465 18114 catalog_manager.cc:4830] TS ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): tablet 5bc67ebb58a246d9a8c47f0838a4831a (table rebalance_test_table_2 [id=1718400b78bf4e67aa0074e6ad933f73]) successfully deleted
I20240503 14:14:52.648285 18639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:52.651429 18510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:52.657510 18248 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:52.673032 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:52.682541 20322 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:52.704090 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:52.726994 20585 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:52.750298 21521 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:52.766897 18378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:52.985332 18639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:52.986804 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:53.002787 18510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:53.018954 20322 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:53.049944 20585 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:53.050115 18248 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:53.079964 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:53.101270 21521 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:53.102234 18378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:53.970045 18510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:53.970186 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:53.981257 18248 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:53.987567 18639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:54.001086 20322 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:54.045722 20585 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:54.071648 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:54.086194 21521 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:54.125069 18378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:54.324388 18639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:54.326318 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:54.335031 18510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:54.337817 20322 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:54.355607 20585 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:54.361898 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:54.370507 18248 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:54.420049 21521 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:54.433599 18378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:54.680464 18639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:54.698359 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:54.704996 18248 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:54.705467 20322 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:54.711817 18510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:54.728737 20585 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:54.776363 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:54.788645 21521 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:54.801600 18378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:55.175299 18639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:55.197618 20322 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:55.212402 20585 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:55.216991 18248 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:55.259867 18378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:55.261262 20190 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:55.261718 20455 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:55.262651 18510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:55.263249 21521 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4127b9518b394f33afa8da3f0c1e0072 | 127.10.145.62:33209 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.145.20:44655 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1c72210d694344d4b4604140088a6471 | 127.10.145.4:36411 | HEALTHY | <none> | 8 | 0
2e6a3f22c7ae444e96f6d35cca602e62 | 127.10.145.5:40605 | HEALTHY | <none> | 0 | 0
55339e38e7aa4bb7b7b42aa206108722 | 127.10.145.3:38857 | HEALTHY | <none> | 6 | 0
69cbbbedfe03437b8c95b7bb730dbe66 | 127.10.145.6:36735 | HEALTHY | <none> | 1 | 0
7ceaaad4bc544c4487859a6671e66838 | 127.10.145.1:33655 | HEALTHY | <none> | 6 | 0
80260781ae364d049c2500ff76a06ce8 | 127.10.145.8:44107 | HEALTHY | <none> | 2 | 0
a632094ba4554a4895d6c0b538b81af8 | 127.10.145.7:38307 | HEALTHY | <none> | 0 | 0
c36ba962bcb34dbea55f4e55689bd38a | 127.10.145.9:45329 | HEALTHY | <none> | 1 | 0
ea366e6e15a24c52b1ea5a154f942568 | 127.10.145.2:43191 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 9
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.10.145.1 | experimental | 127.10.145.1:33655
local_ip_for_outbound_sockets | 127.10.145.2 | experimental | 127.10.145.2:43191
local_ip_for_outbound_sockets | 127.10.145.3 | experimental | 127.10.145.3:38857
local_ip_for_outbound_sockets | 127.10.145.4 | experimental | 127.10.145.4:36411
local_ip_for_outbound_sockets | 127.10.145.5 | experimental | 127.10.145.5:40605
local_ip_for_outbound_sockets | 127.10.145.6 | experimental | 127.10.145.6:36735
local_ip_for_outbound_sockets | 127.10.145.7 | experimental | 127.10.145.7:38307
local_ip_for_outbound_sockets | 127.10.145.8 | experimental | 127.10.145.8:44107
local_ip_for_outbound_sockets | 127.10.145.9 | experimental | 127.10.145.9:45329
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.145.1:33655
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.145.2:43191
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.145.3:38857
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.10.145.4:36411
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.10.145.5:40605
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.10.145.6:36735
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.10.145.7:38307
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.10.145.8:44107
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/info.pb | hidden | 127.10.145.9:45329
tablet_copy_download_file_inject_latency_ms | 1500 | hidden | all 9 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.145.20:44655 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 10
Tablets | 30
Replicas | 90
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:14:55.368729 10820 log_verifier.cc:126] Checking tablet 10a4ca83469d4dab823a26d6cbe952a6
I20240503 14:14:55.406800 10820 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 10a4ca83469d4dab823a26d6cbe952a6
I20240503 14:14:55.406929 10820 log_verifier.cc:126] Checking tablet 19b1b6be3bda47829f53765282eae9f9
I20240503 14:14:55.447924 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 19b1b6be3bda47829f53765282eae9f9
I20240503 14:14:55.448033 10820 log_verifier.cc:126] Checking tablet 3de72df2a82e445f9aa783bc741fa996
I20240503 14:14:55.523140 10820 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3de72df2a82e445f9aa783bc741fa996
I20240503 14:14:55.523238 10820 log_verifier.cc:126] Checking tablet 4cb29bdb71f24b46b32e7b97d0172c20
I20240503 14:14:55.563550 10820 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 4cb29bdb71f24b46b32e7b97d0172c20
I20240503 14:14:55.563669 10820 log_verifier.cc:126] Checking tablet 4d24e31ff3ad41d98214bb41d8c28fda
I20240503 14:14:55.600637 10820 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4d24e31ff3ad41d98214bb41d8c28fda
I20240503 14:14:55.600764 10820 log_verifier.cc:126] Checking tablet 5a23a38ec48148b19bf8102a944ab11e
I20240503 14:14:55.638233 10820 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 5a23a38ec48148b19bf8102a944ab11e
I20240503 14:14:55.638348 10820 log_verifier.cc:126] Checking tablet 5bc67ebb58a246d9a8c47f0838a4831a
I20240503 14:14:55.678546 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 5bc67ebb58a246d9a8c47f0838a4831a
I20240503 14:14:55.678674 10820 log_verifier.cc:126] Checking tablet 71a326db29114366951a67b9aa84ac03
I20240503 14:14:55.716835 10820 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 71a326db29114366951a67b9aa84ac03
I20240503 14:14:55.716965 10820 log_verifier.cc:126] Checking tablet 7570d5bce0e5481d8dded653617be5a7
I20240503 14:14:55.796957 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 7570d5bce0e5481d8dded653617be5a7
I20240503 14:14:55.797091 10820 log_verifier.cc:126] Checking tablet 79373bd79a934eaab67bbcf2208e7b5b
I20240503 14:14:55.840210 10820 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 79373bd79a934eaab67bbcf2208e7b5b
I20240503 14:14:55.840322 10820 log_verifier.cc:126] Checking tablet 7c63b35eef494f0e8928c4ca0f74ec32
I20240503 14:14:55.878427 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 7c63b35eef494f0e8928c4ca0f74ec32
I20240503 14:14:55.878530 10820 log_verifier.cc:126] Checking tablet 7f9705f607a94835ad7e6c20668b80ad
I20240503 14:14:55.916340 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 7f9705f607a94835ad7e6c20668b80ad
I20240503 14:14:55.916451 10820 log_verifier.cc:126] Checking tablet 87fded61bfc240f5846080c328528edc
I20240503 14:14:55.962085 10820 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 87fded61bfc240f5846080c328528edc
I20240503 14:14:55.962185 10820 log_verifier.cc:126] Checking tablet 8c337ef0405e4856bd2a608a6a602a49
I20240503 14:14:56.004906 10820 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 8c337ef0405e4856bd2a608a6a602a49
I20240503 14:14:56.005054 10820 log_verifier.cc:126] Checking tablet 91743575cf4a4f38b5b43265539237a3
I20240503 14:14:56.080646 10820 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 91743575cf4a4f38b5b43265539237a3
I20240503 14:14:56.080756 10820 log_verifier.cc:126] Checking tablet 9577e6d3801f42f89afe8a48a77031dd
I20240503 14:14:56.120306 10820 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 9577e6d3801f42f89afe8a48a77031dd
I20240503 14:14:56.120415 10820 log_verifier.cc:126] Checking tablet 963ecc30d66b43a4ad149f596bd288d9
I20240503 14:14:56.159493 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 963ecc30d66b43a4ad149f596bd288d9
I20240503 14:14:56.159610 10820 log_verifier.cc:126] Checking tablet a39fbf1d8f6f46ccab3c2b992ed0c63e
I20240503 14:14:56.197270 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet a39fbf1d8f6f46ccab3c2b992ed0c63e
I20240503 14:14:56.197420 10820 log_verifier.cc:126] Checking tablet c18cad62ab6443cea66d08f0980c75b6
I20240503 14:14:56.234426 10820 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c18cad62ab6443cea66d08f0980c75b6
I20240503 14:14:56.234530 10820 log_verifier.cc:126] Checking tablet c2167226099a4dcf98304918e8f412af
I20240503 14:14:56.271317 10820 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c2167226099a4dcf98304918e8f412af
I20240503 14:14:56.271427 10820 log_verifier.cc:126] Checking tablet c35376539bc944e59b74bfe59d5feb9e
I20240503 14:14:56.346562 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet c35376539bc944e59b74bfe59d5feb9e
I20240503 14:14:56.346668 10820 log_verifier.cc:126] Checking tablet d5b74777c90f4cb7937d0dfc54420467
I20240503 14:14:56.388840 10820 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d5b74777c90f4cb7937d0dfc54420467
I20240503 14:14:56.388933 10820 log_verifier.cc:126] Checking tablet d907591e3c9740e8b5eee42f8d41996b
I20240503 14:14:56.426864 10820 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d907591e3c9740e8b5eee42f8d41996b
I20240503 14:14:56.426986 10820 log_verifier.cc:126] Checking tablet ea3ea2e3d87f4c099abbf5193c239a8b
I20240503 14:14:56.466723 10820 log_verifier.cc:177] Verified matching terms for 10 ops in tablet ea3ea2e3d87f4c099abbf5193c239a8b
I20240503 14:14:56.466835 10820 log_verifier.cc:126] Checking tablet eeb40a038c4d404d9cb8ea693d9071d7
I20240503 14:14:56.505805 10820 log_verifier.cc:177] Verified matching terms for 10 ops in tablet eeb40a038c4d404d9cb8ea693d9071d7
I20240503 14:14:56.505931 10820 log_verifier.cc:126] Checking tablet eedc3cfb2dff4f9084446d89b9b3c0f4
I20240503 14:14:56.544545 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet eedc3cfb2dff4f9084446d89b9b3c0f4
I20240503 14:14:56.544760 10820 log_verifier.cc:126] Checking tablet f09411ea6c414451b66fd457d9aaf010
I20240503 14:14:56.612675 10820 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f09411ea6c414451b66fd457d9aaf010
I20240503 14:14:56.612773 10820 log_verifier.cc:126] Checking tablet f0c1a7f5739f4fa59271254e501fb095
I20240503 14:14:56.651619 10820 log_verifier.cc:177] Verified matching terms for 10 ops in tablet f0c1a7f5739f4fa59271254e501fb095
I20240503 14:14:56.651751 10820 log_verifier.cc:126] Checking tablet f2c5c6f4317649fab75d01e339b76403
I20240503 14:14:56.691448 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet f2c5c6f4317649fab75d01e339b76403
I20240503 14:14:56.691581 10820 log_verifier.cc:126] Checking tablet fe3d20ad69474bcbae19bdd06cbfeb52
I20240503 14:14:56.729223 10820 log_verifier.cc:177] Verified matching terms for 9 ops in tablet fe3d20ad69474bcbae19bdd06cbfeb52
I20240503 14:14:56.730271 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 18187
I20240503 14:14:56.737248 18309 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:56.978277 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 18187
W20240503 14:14:57.020987 21389 connection.cc:535] server connection from 127.10.145.1:50663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:57.022125 18333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.145.1:33655 (error 108)
I20240503 14:14:57.022257 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 18318
W20240503 14:14:57.024433 18333 consensus_peers.cc:485] T d907591e3c9740e8b5eee42f8d41996b P ea366e6e15a24c52b1ea5a154f942568 -> Peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): Couldn't send request to peer 7ceaaad4bc544c4487859a6671e66838. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:33655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:57.024609 18333 consensus_peers.cc:485] T 87fded61bfc240f5846080c328528edc P ea366e6e15a24c52b1ea5a154f942568 -> Peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): Couldn't send request to peer 7ceaaad4bc544c4487859a6671e66838. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:33655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:57.028664 18438 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:14:57.220346 21389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.1:33655: connect: Connection refused (error 111)
W20240503 14:14:57.221832 21389 consensus_peers.cc:485] T 8c337ef0405e4856bd2a608a6a602a49 P c36ba962bcb34dbea55f4e55689bd38a -> Peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): Couldn't send request to peer 7ceaaad4bc544c4487859a6671e66838. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.1:33655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:57.241139 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 18318
I20240503 14:14:57.273887 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 18448
W20240503 14:14:57.273849 18595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.145.2:43191 (error 108)
W20240503 14:14:57.275815 18595 consensus_peers.cc:485] T f09411ea6c414451b66fd457d9aaf010 P 1c72210d694344d4b4604140088a6471 -> Peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): Couldn't send request to peer ea366e6e15a24c52b1ea5a154f942568. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.2:43191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:57.275943 18595 consensus_peers.cc:485] T c2167226099a4dcf98304918e8f412af P 1c72210d694344d4b4604140088a6471 -> Peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): Couldn't send request to peer ea366e6e15a24c52b1ea5a154f942568. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.2:43191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:57.280231 18569 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:14:57.330281 18595 consensus_peers.cc:485] T 3de72df2a82e445f9aa783bc741fa996 P 1c72210d694344d4b4604140088a6471 -> Peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): Couldn't send request to peer ea366e6e15a24c52b1ea5a154f942568. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.2:43191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:14:57.456174 20540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.2:43191: connect: Connection refused (error 111)
W20240503 14:14:57.457837 20540 consensus_peers.cc:485] T 91743575cf4a4f38b5b43265539237a3 P 80260781ae364d049c2500ff76a06ce8 -> Peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): Couldn't send request to peer ea366e6e15a24c52b1ea5a154f942568. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.2:43191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:57.497627 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 18448
W20240503 14:14:57.530726 20541 connection.cc:535] server connection from 127.10.145.3:39519 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:14:57.530831 20278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.145.3:38857 (error 108)
I20240503 14:14:57.531800 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 18578
W20240503 14:14:57.533030 20278 consensus_peers.cc:485] T f0c1a7f5739f4fa59271254e501fb095 P 69cbbbedfe03437b8c95b7bb730dbe66 -> Peer 55339e38e7aa4bb7b7b42aa206108722 (127.10.145.3:38857): Couldn't send request to peer 55339e38e7aa4bb7b7b42aa206108722. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.3:38857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:57.538360 18699 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:57.755576 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 18578
I20240503 14:14:57.791015 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 20126
I20240503 14:14:57.803186 20249 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:14:57.928820 20538 consensus_peers.cc:485] T ea3ea2e3d87f4c099abbf5193c239a8b P 80260781ae364d049c2500ff76a06ce8 -> Peer 1c72210d694344d4b4604140088a6471 (127.10.145.4:36411): Couldn't send request to peer 1c72210d694344d4b4604140088a6471. Status: Network error: Client connection negotiation failed: client connection to 127.10.145.4:36411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:57.956092 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 20126
I20240503 14:14:57.977880 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 20259
I20240503 14:14:57.984251 20381 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:58.084229 24519 raft_consensus.cc:482] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ceaaad4bc544c4487859a6671e66838)
I20240503 14:14:58.084465 24519 raft_consensus.cc:504] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:58.085522 24519 leader_election.cc:288] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605)
W20240503 14:14:58.087090 21392 leader_election.cc:334] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Network error: Client connection negotiation failed: client connection to 127.10.145.5:40605: connect: Connection refused (error 111)
W20240503 14:14:58.087288 21389 leader_election.cc:334] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): Network error: Client connection negotiation failed: client connection to 127.10.145.1:33655: connect: Connection refused (error 111)
I20240503 14:14:58.087433 21389 leader_election.cc:302] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c36ba962bcb34dbea55f4e55689bd38a; no voters: 2e6a3f22c7ae444e96f6d35cca602e62, 7ceaaad4bc544c4487859a6671e66838
I20240503 14:14:58.087695 24519 raft_consensus.cc:2726] T fe3d20ad69474bcbae19bdd06cbfeb52 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:58.094554 24531 raft_consensus.cc:482] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea366e6e15a24c52b1ea5a154f942568)
I20240503 14:14:58.094771 24531 raft_consensus.cc:504] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "a632094ba4554a4895d6c0b538b81af8" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 38307 } attrs { promote: false } }
I20240503 14:14:58.096336 24531 leader_election.cc:288] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191)
W20240503 14:14:58.097518 20407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.145.1:33655: connect: Connection refused (error 111)
W20240503 14:14:58.098836 20409 leader_election.cc:334] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): Network error: Client connection negotiation failed: client connection to 127.10.145.2:43191: connect: Connection refused (error 111)
W20240503 14:14:58.099032 20407 leader_election.cc:334] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): Network error: Client connection negotiation failed: client connection to 127.10.145.1:33655: connect: Connection refused (error 111)
I20240503 14:14:58.099233 20407 leader_election.cc:302] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a632094ba4554a4895d6c0b538b81af8; no voters: 7ceaaad4bc544c4487859a6671e66838, ea366e6e15a24c52b1ea5a154f942568
I20240503 14:14:58.099596 24531 raft_consensus.cc:2726] T 87fded61bfc240f5846080c328528edc P a632094ba4554a4895d6c0b538b81af8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:58.135421 24525 raft_consensus.cc:482] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ceaaad4bc544c4487859a6671e66838)
I20240503 14:14:58.135658 24525 raft_consensus.cc:504] T a39fbf1d8f6f46ccab3c2b992ed0c63e P 69cbbbedfe03437b8c95b7bb730dbe66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:58.136057 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 20259
I20240503 14:14:58.157871 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 20391
I20240503 14:14:58.163887 20514 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:58.306494 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 20391
I20240503 14:14:58.320654 24519 raft_consensus.cc:482] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ea366e6e15a24c52b1ea5a154f942568)
I20240503 14:14:58.320956 24519 raft_consensus.cc:504] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:58.321875 24519 leader_election.cc:288] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605)
W20240503 14:14:58.323302 21391 leader_election.cc:334] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): Network error: Client connection negotiation failed: client connection to 127.10.145.2:43191: connect: Connection refused (error 111)
W20240503 14:14:58.323527 21392 leader_election.cc:334] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Network error: Client connection negotiation failed: client connection to 127.10.145.5:40605: connect: Connection refused (error 111)
I20240503 14:14:58.323663 21392 leader_election.cc:302] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c36ba962bcb34dbea55f4e55689bd38a; no voters: 2e6a3f22c7ae444e96f6d35cca602e62, ea366e6e15a24c52b1ea5a154f942568
I20240503 14:14:58.323925 24519 raft_consensus.cc:2726] T eeb40a038c4d404d9cb8ea693d9071d7 P c36ba962bcb34dbea55f4e55689bd38a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:58.328938 18115 connection.cc:535] server connection from 127.10.145.7:35109 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:14:58.329170 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 20523
I20240503 14:14:58.330539 24526 raft_consensus.cc:482] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea366e6e15a24c52b1ea5a154f942568)
I20240503 14:14:58.330771 24526 raft_consensus.cc:504] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:58.331914 24526 leader_election.cc:288] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191), 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735)
W20240503 14:14:58.333384 20540 leader_election.cc:334] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ea366e6e15a24c52b1ea5a154f942568 (127.10.145.2:43191): Network error: Client connection negotiation failed: client connection to 127.10.145.2:43191: connect: Connection refused (error 111)
W20240503 14:14:58.333624 20539 leader_election.cc:334] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Network error: Client connection negotiation failed: client connection to 127.10.145.6:36735: connect: Connection refused (error 111)
I20240503 14:14:58.333729 20539 leader_election.cc:302] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80260781ae364d049c2500ff76a06ce8; no voters: 69cbbbedfe03437b8c95b7bb730dbe66, ea366e6e15a24c52b1ea5a154f942568
I20240503 14:14:58.334014 24526 raft_consensus.cc:2726] T 963ecc30d66b43a4ad149f596bd288d9 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:58.336565 20643 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:58.479532 24519 raft_consensus.cc:482] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ceaaad4bc544c4487859a6671e66838)
I20240503 14:14:58.479801 24519 raft_consensus.cc:504] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "2e6a3f22c7ae444e96f6d35cca602e62" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 40605 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:58.480190 24539 raft_consensus.cc:482] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ceaaad4bc544c4487859a6671e66838)
I20240503 14:14:58.480377 24539 raft_consensus.cc:504] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7ceaaad4bc544c4487859a6671e66838" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 33655 } } peers { permanent_uuid: "69cbbbedfe03437b8c95b7bb730dbe66" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36735 } attrs { promote: false } } peers { permanent_uuid: "c36ba962bcb34dbea55f4e55689bd38a" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 45329 } attrs { promote: false } }
I20240503 14:14:58.480957 24519 leader_election.cc:288] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605)
I20240503 14:14:58.481420 24539 leader_election.cc:288] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655), 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735)
W20240503 14:14:58.482605 21392 leader_election.cc:334] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2e6a3f22c7ae444e96f6d35cca602e62 (127.10.145.5:40605): Network error: Client connection negotiation failed: client connection to 127.10.145.5:40605: connect: Connection refused (error 111)
W20240503 14:14:58.482787 21389 leader_election.cc:334] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): Network error: Client connection negotiation failed: client connection to 127.10.145.1:33655: connect: Connection refused (error 111)
I20240503 14:14:58.482924 21389 leader_election.cc:302] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c36ba962bcb34dbea55f4e55689bd38a; no voters: 2e6a3f22c7ae444e96f6d35cca602e62, 7ceaaad4bc544c4487859a6671e66838
I20240503 14:14:58.483225 24539 raft_consensus.cc:2726] T f2c5c6f4317649fab75d01e339b76403 P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:58.483259 21389 leader_election.cc:334] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ceaaad4bc544c4487859a6671e66838 (127.10.145.1:33655): Network error: Client connection negotiation failed: client connection to 127.10.145.1:33655: connect: Connection refused (error 111)
W20240503 14:14:58.483443 21390 leader_election.cc:334] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 69cbbbedfe03437b8c95b7bb730dbe66 (127.10.145.6:36735): Network error: Client connection negotiation failed: client connection to 127.10.145.6:36735: connect: Connection refused (error 111)
I20240503 14:14:58.483587 21390 leader_election.cc:302] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c36ba962bcb34dbea55f4e55689bd38a; no voters: 69cbbbedfe03437b8c95b7bb730dbe66, 7ceaaad4bc544c4487859a6671e66838
I20240503 14:14:58.483824 24539 raft_consensus.cc:2726] T a39fbf1d8f6f46ccab3c2b992ed0c63e P c36ba962bcb34dbea55f4e55689bd38a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:58.489624 24482 raft_consensus.cc:482] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c72210d694344d4b4604140088a6471)
I20240503 14:14:58.489845 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 20523
I20240503 14:14:58.489840 24482 raft_consensus.cc:504] T 3de72df2a82e445f9aa783bc741fa996 P 80260781ae364d049c2500ff76a06ce8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea366e6e15a24c52b1ea5a154f942568" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 43191 } } peers { permanent_uuid: "1c72210d694344d4b4604140088a6471" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 36411 } } peers { permanent_uuid: "80260781ae364d049c2500ff76a06ce8" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44107 } attrs { promote: false } }
I20240503 14:14:58.512090 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 21280
I20240503 14:14:58.518185 21628 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:58.646639 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 21280
I20240503 14:14:58.666232 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 18097
I20240503 14:14:58.667302 18157 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:14:58.879626 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 18097
2024-05-03T14:14:58Z chronyd exiting
[ OK ] TserverAddedDuringRebalancingTest.TserverStarts/1 (52653 ms)
[----------] 1 test from TserverAddedDuringRebalancingTest (52653 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5
I20240503 14:14:59.031623 10820 test_util.cc:268] Using random seed: 1480940066
I20240503 14:14:59.033640 10820 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:14:59.033731 10820 ts_itest-base.cc:113] --------------
I20240503 14:14:59.033772 10820 ts_itest-base.cc:114] 4 tablet servers
I20240503 14:14:59.033838 10820 ts_itest-base.cc:115] 1 replicas per TS
I20240503 14:14:59.033879 10820 ts_itest-base.cc:116] --------------
2024-05-03T14:14:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:14:59Z Disabled control of system clock
I20240503 14:14:59.085227 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:42931
--webserver_interface=127.10.145.62
--webserver_port=0
--builtin_ntp_servers=127.10.145.20:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.145.62:42931
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 14:14:59.432463 24550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:14:59.432813 24550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:14:59.432965 24550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:14:59.440582 24550 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:14:59.440749 24550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:14:59.440833 24550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:14:59.440907 24550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:14:59.440975 24550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:14:59.450490 24550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.10.145.62:42931
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:42931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:14:59.451875 24550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:14:59.453780 24550 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:14:59.465236 24556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:59.465173 24555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:14:59.465632 24558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:14:59.466076 24550 server_base.cc:1034] running on GCE node
I20240503 14:14:59.466897 24550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:14:59.468442 24550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:14:59.469664 24550 hybrid_clock.cc:648] HybridClock initialized: now 1714745699469612 us; error 43 us; skew 500 ppm
I20240503 14:14:59.470050 24550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:14:59.472568 24550 webserver.cc:460] Webserver started at http://127.10.145.62:34103/ using document root <none> and password file <none>
I20240503 14:14:59.473455 24550 fs_manager.cc:359] Metadata directory not provided
I20240503 14:14:59.473587 24550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:14:59.473910 24550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:14:59.476686 24550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7c8e126a71a341c7ab77674a53ab2f7a"
format_stamp: "Formatted at 2024-05-03 14:14:59 on dist-test-slave-v549"
I20240503 14:14:59.477563 24550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7c8e126a71a341c7ab77674a53ab2f7a"
format_stamp: "Formatted at 2024-05-03 14:14:59 on dist-test-slave-v549"
I20240503 14:14:59.483275 24550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:14:59.487303 24564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:14:59.488824 24550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:59.489032 24550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7c8e126a71a341c7ab77674a53ab2f7a"
format_stamp: "Formatted at 2024-05-03 14:14:59 on dist-test-slave-v549"
I20240503 14:14:59.489269 24550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:14:59.523339 24550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:14:59.524266 24550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:14:59.524580 24550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:14:59.542671 24550 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.62:42931
I20240503 14:14:59.542707 24615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.62:42931 every 8 connection(s)
I20240503 14:14:59.544129 24550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:14:59.545835 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 24550
I20240503 14:14:59.546110 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:14:59.549569 24616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:59.564533 24616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a: Bootstrap starting.
I20240503 14:14:59.569626 24616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a: Neither blocks nor log segments found. Creating new log.
I20240503 14:14:59.571146 24616 log.cc:824] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a: Log is configured to *not* fsync() on all Append() calls
I20240503 14:14:59.575008 24616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a: No bootstrap required, opened a new log
I20240503 14:14:59.586284 24616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8e126a71a341c7ab77674a53ab2f7a" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42931 } }
I20240503 14:14:59.586895 24616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:14:59.587021 24616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8e126a71a341c7ab77674a53ab2f7a, State: Initialized, Role: FOLLOWER
I20240503 14:14:59.587957 24616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8e126a71a341c7ab77674a53ab2f7a" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42931 } }
I20240503 14:14:59.588279 24616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:14:59.588430 24616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:14:59.588611 24616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:14:59.592003 24616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8e126a71a341c7ab77674a53ab2f7a" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42931 } }
I20240503 14:14:59.592800 24616 leader_election.cc:302] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c8e126a71a341c7ab77674a53ab2f7a; no voters:
I20240503 14:14:59.593755 24616 leader_election.cc:288] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:14:59.593869 24619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:14:59.596341 24616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:14:59.600711 24619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [term 1 LEADER]: Becoming Leader. State: Replica: 7c8e126a71a341c7ab77674a53ab2f7a, State: Running, Role: LEADER
I20240503 14:14:59.601429 24619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8e126a71a341c7ab77674a53ab2f7a" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42931 } }
I20240503 14:14:59.605859 24620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c8e126a71a341c7ab77674a53ab2f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8e126a71a341c7ab77674a53ab2f7a" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42931 } } }
I20240503 14:14:59.606220 24620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [sys.catalog]: This master's current role is: LEADER
I20240503 14:14:59.606711 24621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c8e126a71a341c7ab77674a53ab2f7a. Latest consensus state: current_term: 1 leader_uuid: "7c8e126a71a341c7ab77674a53ab2f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8e126a71a341c7ab77674a53ab2f7a" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42931 } } }
I20240503 14:14:59.607038 24621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a [sys.catalog]: This master's current role is: LEADER
I20240503 14:14:59.611467 24633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:14:59.617092 24633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:14:59.629155 24633 catalog_manager.cc:1347] Generated new cluster ID: 9293ceb3b43e407ca7193fa893083198
I20240503 14:14:59.629336 24633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:14:59.645133 24633 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:14:59.646682 24633 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:14:59.666672 24633 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7c8e126a71a341c7ab77674a53ab2f7a: Generated new TSK 0
I20240503 14:14:59.667896 24633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:14:59.679358 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.1:0
--local_ip_for_outbound_sockets=127.10.145.1
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42931
--builtin_ntp_servers=127.10.145.20:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:15:00.031917 24641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:00.032236 24641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:00.032374 24641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:00.040020 24641 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:00.040159 24641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:00.040328 24641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.1
I20240503 14:15:00.049634 24641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42931
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.1
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:00.051033 24641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:00.053197 24641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:00.065680 24647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:00.065984 24649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:00.066296 24646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:00.066711 24641 server_base.cc:1034] running on GCE node
I20240503 14:15:00.067333 24641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:00.068320 24641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:00.069522 24641 hybrid_clock.cc:648] HybridClock initialized: now 1714745700069462 us; error 53 us; skew 500 ppm
I20240503 14:15:00.069854 24641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:00.072166 24641 webserver.cc:460] Webserver started at http://127.10.145.1:40977/ using document root <none> and password file <none>
I20240503 14:15:00.073032 24641 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:00.073158 24641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:00.073515 24641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:00.076231 24641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3f3cf1cac8e24dcc972a6d194f1903ab"
format_stamp: "Formatted at 2024-05-03 14:15:00 on dist-test-slave-v549"
I20240503 14:15:00.077051 24641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3f3cf1cac8e24dcc972a6d194f1903ab"
format_stamp: "Formatted at 2024-05-03 14:15:00 on dist-test-slave-v549"
I20240503 14:15:00.082655 24641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 14:15:00.086421 24655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:00.087962 24641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:15:00.088141 24641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3f3cf1cac8e24dcc972a6d194f1903ab"
format_stamp: "Formatted at 2024-05-03 14:15:00 on dist-test-slave-v549"
I20240503 14:15:00.088358 24641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:00.104991 24641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:00.105937 24641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:00.106210 24641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:00.107197 24641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:00.108788 24641 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:00.108878 24641 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:00.108992 24641 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:00.109066 24641 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:00.139711 24641 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.1:37737
I20240503 14:15:00.139760 24767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.1:37737 every 8 connection(s)
I20240503 14:15:00.141070 24641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:15:00.145247 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 24641
I20240503 14:15:00.145529 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:15:00.149171 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.2:0
--local_ip_for_outbound_sockets=127.10.145.2
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42931
--builtin_ntp_servers=127.10.145.20:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:00.155382 24768 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42931
I20240503 14:15:00.155781 24768 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:00.156862 24768 heartbeater.cc:511] Master 127.10.145.62:42931 requested a full tablet report, sending...
I20240503 14:15:00.159245 24581 ts_manager.cc:194] Registered new tserver with Master: 3f3cf1cac8e24dcc972a6d194f1903ab (127.10.145.1:37737)
I20240503 14:15:00.161398 24581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.1:57315
W20240503 14:15:00.505625 24772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:00.505940 24772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:00.506031 24772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:00.514027 24772 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:00.514180 24772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:00.514300 24772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.2
I20240503 14:15:00.523998 24772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42931
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.2
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:00.525501 24772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:00.527366 24772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:00.540115 24778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:00.540508 24780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:00.540879 24777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:00.541455 24772 server_base.cc:1034] running on GCE node
I20240503 14:15:00.542209 24772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:00.543556 24772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:00.544807 24772 hybrid_clock.cc:648] HybridClock initialized: now 1714745700544730 us; error 67 us; skew 500 ppm
I20240503 14:15:00.545224 24772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:00.548291 24772 webserver.cc:460] Webserver started at http://127.10.145.2:38711/ using document root <none> and password file <none>
I20240503 14:15:00.549377 24772 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:00.549546 24772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:00.549930 24772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:00.553685 24772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ce339dd856e343b19543edbb616974f3"
format_stamp: "Formatted at 2024-05-03 14:15:00 on dist-test-slave-v549"
I20240503 14:15:00.554848 24772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ce339dd856e343b19543edbb616974f3"
format_stamp: "Formatted at 2024-05-03 14:15:00 on dist-test-slave-v549"
I20240503 14:15:00.562207 24772 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20240503 14:15:00.566027 24786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:00.567582 24772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:00.567768 24772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ce339dd856e343b19543edbb616974f3"
format_stamp: "Formatted at 2024-05-03 14:15:00 on dist-test-slave-v549"
I20240503 14:15:00.567992 24772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:00.585773 24772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:00.586730 24772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:00.587033 24772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:00.588068 24772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:00.589690 24772 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:00.589792 24772 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:00.589915 24772 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:00.589987 24772 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:00.621220 24772 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.2:45183
I20240503 14:15:00.621325 24898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.2:45183 every 8 connection(s)
I20240503 14:15:00.622643 24772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:15:00.623486 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 24772
I20240503 14:15:00.623693 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:15:00.627580 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.3:0
--local_ip_for_outbound_sockets=127.10.145.3
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42931
--builtin_ntp_servers=127.10.145.20:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:00.639066 24899 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42931
I20240503 14:15:00.639457 24899 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:00.640587 24899 heartbeater.cc:511] Master 127.10.145.62:42931 requested a full tablet report, sending...
I20240503 14:15:00.642535 24581 ts_manager.cc:194] Registered new tserver with Master: ce339dd856e343b19543edbb616974f3 (127.10.145.2:45183)
I20240503 14:15:00.643364 24581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.2:55487
W20240503 14:15:00.985838 24902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:00.986181 24902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:00.986306 24902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:00.994134 24902 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:00.994290 24902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:00.994444 24902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.3
I20240503 14:15:01.003985 24902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42931
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.3
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:01.005426 24902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:01.007335 24902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:01.020356 24908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:01.020555 24910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:01.020844 24907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:01.021411 24902 server_base.cc:1034] running on GCE node
I20240503 14:15:01.022172 24902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:01.023228 24902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:01.024449 24902 hybrid_clock.cc:648] HybridClock initialized: now 1714745701024394 us; error 51 us; skew 500 ppm
I20240503 14:15:01.024807 24902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:01.027446 24902 webserver.cc:460] Webserver started at http://127.10.145.3:44673/ using document root <none> and password file <none>
I20240503 14:15:01.028255 24902 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:01.028414 24902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:01.028884 24902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:01.031898 24902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "35d33222882b409cbda9c6b44847500d"
format_stamp: "Formatted at 2024-05-03 14:15:01 on dist-test-slave-v549"
I20240503 14:15:01.032902 24902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "35d33222882b409cbda9c6b44847500d"
format_stamp: "Formatted at 2024-05-03 14:15:01 on dist-test-slave-v549"
I20240503 14:15:01.038820 24902 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20240503 14:15:01.042706 24916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:01.044273 24902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:01.044456 24902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "35d33222882b409cbda9c6b44847500d"
format_stamp: "Formatted at 2024-05-03 14:15:01 on dist-test-slave-v549"
I20240503 14:15:01.044689 24902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:01.071666 24902 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:01.072669 24902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:01.073010 24902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:01.074115 24902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:01.075768 24902 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:01.075882 24902 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:01.075996 24902 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:01.076068 24902 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:01.107435 24902 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.3:46591
I20240503 14:15:01.107474 25028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.3:46591 every 8 connection(s)
I20240503 14:15:01.108796 24902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:15:01.113837 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 24902
I20240503 14:15:01.114002 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:15:01.117034 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.4:0
--local_ip_for_outbound_sockets=127.10.145.4
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42931
--builtin_ntp_servers=127.10.145.20:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:01.122116 25029 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42931
I20240503 14:15:01.122471 25029 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:01.123558 25029 heartbeater.cc:511] Master 127.10.145.62:42931 requested a full tablet report, sending...
I20240503 14:15:01.125319 24581 ts_manager.cc:194] Registered new tserver with Master: 35d33222882b409cbda9c6b44847500d (127.10.145.3:46591)
I20240503 14:15:01.126390 24581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.3:43001
I20240503 14:15:01.164974 24768 heartbeater.cc:503] Master 127.10.145.62:42931 was elected leader, sending a full tablet report...
W20240503 14:15:01.472937 25032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:01.473316 25032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:01.473409 25032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:01.481109 25032 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:01.481249 25032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:01.481415 25032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.4
I20240503 14:15:01.490813 25032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42931
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.4
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:01.492187 25032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:01.494207 25032 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:01.507076 25038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:01.507225 25037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:01.507772 25040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:01.507999 25032 server_base.cc:1034] running on GCE node
I20240503 14:15:01.508652 25032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:01.509758 25032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:01.510969 25032 hybrid_clock.cc:648] HybridClock initialized: now 1714745701510916 us; error 46 us; skew 500 ppm
I20240503 14:15:01.511328 25032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:01.513919 25032 webserver.cc:460] Webserver started at http://127.10.145.4:46357/ using document root <none> and password file <none>
I20240503 14:15:01.514740 25032 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:01.514870 25032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:01.515202 25032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:01.518110 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0709ec44591a468781515e7157d6b1e5"
format_stamp: "Formatted at 2024-05-03 14:15:01 on dist-test-slave-v549"
I20240503 14:15:01.518996 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0709ec44591a468781515e7157d6b1e5"
format_stamp: "Formatted at 2024-05-03 14:15:01 on dist-test-slave-v549"
I20240503 14:15:01.524828 25032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:15:01.528610 25046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:01.530251 25032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:01.530447 25032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0709ec44591a468781515e7157d6b1e5"
format_stamp: "Formatted at 2024-05-03 14:15:01 on dist-test-slave-v549"
I20240503 14:15:01.530663 25032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:01.555752 25032 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:01.556699 25032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:01.556984 25032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:01.558086 25032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:01.559731 25032 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:01.559827 25032 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:01.559952 25032 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:01.560020 25032 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:01.591682 25032 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.4:39503
I20240503 14:15:01.591734 25158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.4:39503 every 8 connection(s)
I20240503 14:15:01.593118 25032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:15:01.600343 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 25032
I20240503 14:15:01.600538 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:15:01.606598 25159 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42931
I20240503 14:15:01.606897 25159 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:01.607756 25159 heartbeater.cc:511] Master 127.10.145.62:42931 requested a full tablet report, sending...
I20240503 14:15:01.609200 24581 ts_manager.cc:194] Registered new tserver with Master: 0709ec44591a468781515e7157d6b1e5 (127.10.145.4:39503)
I20240503 14:15:01.610028 24581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.4:41497
I20240503 14:15:01.617265 10820 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 14:15:01.633664 24581 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58294:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:15:01.646263 24899 heartbeater.cc:503] Master 127.10.145.62:42931 was elected leader, sending a full tablet report...
I20240503 14:15:01.662055 24964 tablet_service.cc:1467] Processing CreateTablet for tablet 59f53dd9e5bc4475b314a98c85f7a7eb (DEFAULT_TABLE table=TestTable [id=d4aee74bba024191a46fcf3f07638435]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:01.663439 24964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f53dd9e5bc4475b314a98c85f7a7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:01.673980 25173 tablet_bootstrap.cc:492] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Bootstrap starting.
I20240503 14:15:01.677390 25173 tablet_bootstrap.cc:654] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:01.678507 25173 log.cc:824] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:01.681121 25173 tablet_bootstrap.cc:492] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: No bootstrap required, opened a new log
I20240503 14:15:01.681453 25173 ts_tablet_manager.cc:1367] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240503 14:15:01.687985 25173 raft_consensus.cc:348] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d33222882b409cbda9c6b44847500d" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46591 } }
I20240503 14:15:01.688381 25173 raft_consensus.cc:374] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:01.688517 25173 raft_consensus.cc:729] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d33222882b409cbda9c6b44847500d, State: Initialized, Role: FOLLOWER
I20240503 14:15:01.689189 25173 consensus_queue.cc:260] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d33222882b409cbda9c6b44847500d" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46591 } }
I20240503 14:15:01.689460 25173 raft_consensus.cc:388] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:01.689599 25173 raft_consensus.cc:482] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:01.689752 25173 raft_consensus.cc:3037] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:01.692035 25173 raft_consensus.cc:504] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d33222882b409cbda9c6b44847500d" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46591 } }
I20240503 14:15:01.692685 25173 leader_election.cc:302] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35d33222882b409cbda9c6b44847500d; no voters:
I20240503 14:15:01.693351 25173 leader_election.cc:288] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:01.693643 25175 raft_consensus.cc:2781] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:01.694240 25175 raft_consensus.cc:686] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 1 LEADER]: Becoming Leader. State: Replica: 35d33222882b409cbda9c6b44847500d, State: Running, Role: LEADER
I20240503 14:15:01.694576 25029 heartbeater.cc:503] Master 127.10.145.62:42931 was elected leader, sending a full tablet report...
I20240503 14:15:01.694859 25175 consensus_queue.cc:237] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d33222882b409cbda9c6b44847500d" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46591 } }
I20240503 14:15:01.695713 25173 ts_tablet_manager.cc:1398] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20240503 14:15:01.700966 24580 catalog_manager.cc:5484] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d reported cstate change: term changed from 0 to 1, leader changed from <none> to 35d33222882b409cbda9c6b44847500d (127.10.145.3). New cstate: current_term: 1 leader_uuid: "35d33222882b409cbda9c6b44847500d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d33222882b409cbda9c6b44847500d" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46591 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:01.722093 10820 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 14:15:01.724165 10820 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3f3cf1cac8e24dcc972a6d194f1903ab to finish bootstrapping
I20240503 14:15:01.734066 10820 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ce339dd856e343b19543edbb616974f3 to finish bootstrapping
I20240503 14:15:01.743321 10820 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 35d33222882b409cbda9c6b44847500d to finish bootstrapping
I20240503 14:15:01.751132 10820 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0709ec44591a468781515e7157d6b1e5 to finish bootstrapping
I20240503 14:15:01.761070 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 24902
I20240503 14:15:01.762159 25024 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:01.848976 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 24902
I20240503 14:15:01.863951 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 25032
I20240503 14:15:01.865325 25154 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:01.946250 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 25032
I20240503 14:15:05.712463 24580 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58294:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:05.735287 24703 tablet_service.cc:1467] Processing CreateTablet for tablet d61d8fea94b748f1a62e9ade509782a2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7aa7e6a2dc564760a4a1a729e77bd6cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:05.735239 24834 tablet_service.cc:1467] Processing CreateTablet for tablet 3b60eae0eb9447c79a498d3d13427da3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7aa7e6a2dc564760a4a1a729e77bd6cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:05.735247 24702 tablet_service.cc:1467] Processing CreateTablet for tablet b25a4c1300a148bfa4259093428dbd85 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7aa7e6a2dc564760a4a1a729e77bd6cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:05.736855 24703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61d8fea94b748f1a62e9ade509782a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:05.736864 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b60eae0eb9447c79a498d3d13427da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:05.737339 24702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b25a4c1300a148bfa4259093428dbd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:05.749367 25194 tablet_bootstrap.cc:492] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab: Bootstrap starting.
I20240503 14:15:05.750042 25195 tablet_bootstrap.cc:492] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3: Bootstrap starting.
I20240503 14:15:05.752905 25195 tablet_bootstrap.cc:654] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:05.752907 25194 tablet_bootstrap.cc:654] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:05.754062 25195 log.cc:824] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:05.754061 25194 log.cc:824] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:05.756651 25195 tablet_bootstrap.cc:492] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3: No bootstrap required, opened a new log
I20240503 14:15:05.756664 25194 tablet_bootstrap.cc:492] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab: No bootstrap required, opened a new log
I20240503 14:15:05.756916 25195 ts_tablet_manager.cc:1367] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 14:15:05.756932 25194 ts_tablet_manager.cc:1367] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 14:15:05.763303 25195 raft_consensus.cc:348] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:05.763299 25194 raft_consensus.cc:348] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:05.763666 25194 raft_consensus.cc:374] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:05.763672 25195 raft_consensus.cc:374] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:05.763772 25195 raft_consensus.cc:729] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce339dd856e343b19543edbb616974f3, State: Initialized, Role: FOLLOWER
I20240503 14:15:05.763778 25194 raft_consensus.cc:729] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Initialized, Role: FOLLOWER
I20240503 14:15:05.764398 25194 consensus_queue.cc:260] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:05.764390 25195 consensus_queue.cc:260] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:05.764603 25195 raft_consensus.cc:388] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:05.764601 25194 raft_consensus.cc:388] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:05.764726 25195 raft_consensus.cc:482] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:05.764730 25194 raft_consensus.cc:482] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:05.764870 25194 raft_consensus.cc:3037] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:05.764868 25195 raft_consensus.cc:3037] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:05.767081 25195 raft_consensus.cc:504] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:05.767122 25194 raft_consensus.cc:504] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:05.767716 25195 leader_election.cc:302] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce339dd856e343b19543edbb616974f3; no voters:
I20240503 14:15:05.767711 25194 leader_election.cc:302] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3cf1cac8e24dcc972a6d194f1903ab; no voters:
I20240503 14:15:05.768414 25194 leader_election.cc:288] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:05.768424 25195 leader_election.cc:288] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:05.768548 25198 raft_consensus.cc:2781] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:05.768560 25199 raft_consensus.cc:2781] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:05.774152 25194 ts_tablet_manager.cc:1398] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20240503 14:15:05.774842 25194 tablet_bootstrap.cc:492] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab: Bootstrap starting.
I20240503 14:15:05.775868 25198 raft_consensus.cc:686] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 LEADER]: Becoming Leader. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Running, Role: LEADER
I20240503 14:15:05.776782 25194 tablet_bootstrap.cc:654] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:05.777168 25199 raft_consensus.cc:686] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [term 1 LEADER]: Becoming Leader. State: Replica: ce339dd856e343b19543edbb616974f3, State: Running, Role: LEADER
I20240503 14:15:05.776638 25198 consensus_queue.cc:237] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:05.779570 25199 consensus_queue.cc:237] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:05.780869 25195 ts_tablet_manager.cc:1398] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3: Time spent starting tablet: real 0.024s user 0.011s sys 0.013s
I20240503 14:15:05.785259 24580 catalog_manager.cc:5484] T 3b60eae0eb9447c79a498d3d13427da3 P ce339dd856e343b19543edbb616974f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce339dd856e343b19543edbb616974f3 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ce339dd856e343b19543edbb616974f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:05.788996 25194 tablet_bootstrap.cc:492] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab: No bootstrap required, opened a new log
I20240503 14:15:05.789014 24581 catalog_manager.cc:5484] T d61d8fea94b748f1a62e9ade509782a2 P 3f3cf1cac8e24dcc972a6d194f1903ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3cf1cac8e24dcc972a6d194f1903ab (127.10.145.1). New cstate: current_term: 1 leader_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:05.789232 25194 ts_tablet_manager.cc:1367] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240503 14:15:05.790298 25194 raft_consensus.cc:348] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:05.790537 25194 raft_consensus.cc:374] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:05.790675 25194 raft_consensus.cc:729] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Initialized, Role: FOLLOWER
I20240503 14:15:05.790916 25194 consensus_queue.cc:260] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:05.791141 25194 raft_consensus.cc:388] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:05.791250 25194 raft_consensus.cc:482] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:05.791388 25194 raft_consensus.cc:3037] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:05.794200 25194 raft_consensus.cc:504] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:05.794565 25194 leader_election.cc:302] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3cf1cac8e24dcc972a6d194f1903ab; no voters:
I20240503 14:15:05.794791 25194 leader_election.cc:288] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:05.794893 25202 raft_consensus.cc:2781] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:05.795322 25194 ts_tablet_manager.cc:1398] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:15:05.795565 25202 raft_consensus.cc:686] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 LEADER]: Becoming Leader. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Running, Role: LEADER
I20240503 14:15:05.795828 25202 consensus_queue.cc:237] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:05.798697 24581 catalog_manager.cc:5484] T b25a4c1300a148bfa4259093428dbd85 P 3f3cf1cac8e24dcc972a6d194f1903ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3cf1cac8e24dcc972a6d194f1903ab (127.10.145.1). New cstate: current_term: 1 leader_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.349186 24581 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58294:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:06.358070 24834 tablet_service.cc:1467] Processing CreateTablet for tablet 2afd5a389b1d4639b05a2b04baec64d1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d9571048dea34f09b85b8d70708e40b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:06.358460 24702 tablet_service.cc:1467] Processing CreateTablet for tablet 36b48c566dcb445e97af8caaf6d4fe16 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d9571048dea34f09b85b8d70708e40b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:06.358567 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2afd5a389b1d4639b05a2b04baec64d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:06.358788 24702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b48c566dcb445e97af8caaf6d4fe16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:06.358103 24833 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9242d41b414c6eb03c3fd3c43bad9b (DEFAULT_TABLE table=rebalance_test_table_1 [id=d9571048dea34f09b85b8d70708e40b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:06.361140 24833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b9242d41b414c6eb03c3fd3c43bad9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:06.364058 25234 tablet_bootstrap.cc:492] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3: Bootstrap starting.
I20240503 14:15:06.365306 25235 tablet_bootstrap.cc:492] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab: Bootstrap starting.
I20240503 14:15:06.366318 25234 tablet_bootstrap.cc:654] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:06.366986 25235 tablet_bootstrap.cc:654] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:06.368510 25234 tablet_bootstrap.cc:492] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3: No bootstrap required, opened a new log
I20240503 14:15:06.368654 25234 ts_tablet_manager.cc:1367] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:15:06.368870 25235 tablet_bootstrap.cc:492] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab: No bootstrap required, opened a new log
I20240503 14:15:06.369011 25235 ts_tablet_manager.cc:1367] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:15:06.369550 25234 raft_consensus.cc:348] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.369705 25234 raft_consensus.cc:374] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:06.369657 25235 raft_consensus.cc:348] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.369800 25234 raft_consensus.cc:729] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce339dd856e343b19543edbb616974f3, State: Initialized, Role: FOLLOWER
I20240503 14:15:06.369820 25235 raft_consensus.cc:374] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:06.369907 25235 raft_consensus.cc:729] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Initialized, Role: FOLLOWER
I20240503 14:15:06.370066 25234 consensus_queue.cc:260] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.370122 25235 consensus_queue.cc:260] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.370225 25234 raft_consensus.cc:388] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:06.370301 25234 raft_consensus.cc:482] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:06.370306 25235 raft_consensus.cc:388] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:06.370376 25235 raft_consensus.cc:482] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:06.370402 25234 raft_consensus.cc:3037] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:06.370477 25235 raft_consensus.cc:3037] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:06.372390 25234 raft_consensus.cc:504] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.372543 25235 raft_consensus.cc:504] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.372640 25234 leader_election.cc:302] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce339dd856e343b19543edbb616974f3; no voters:
I20240503 14:15:06.372803 25235 leader_election.cc:302] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3cf1cac8e24dcc972a6d194f1903ab; no voters:
I20240503 14:15:06.373031 25235 leader_election.cc:288] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:06.373162 25202 raft_consensus.cc:2781] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:06.374102 25235 ts_tablet_manager.cc:1398] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20240503 14:15:06.374267 25234 leader_election.cc:288] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:06.374290 25238 raft_consensus.cc:2781] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:06.374584 25202 raft_consensus.cc:686] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 LEADER]: Becoming Leader. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Running, Role: LEADER
I20240503 14:15:06.374861 25202 consensus_queue.cc:237] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.375174 25234 ts_tablet_manager.cc:1398] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 14:15:06.375376 25238 raft_consensus.cc:686] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [term 1 LEADER]: Becoming Leader. State: Replica: ce339dd856e343b19543edbb616974f3, State: Running, Role: LEADER
I20240503 14:15:06.375799 25238 consensus_queue.cc:237] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.375480 25234 tablet_bootstrap.cc:492] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3: Bootstrap starting.
I20240503 14:15:06.377532 24581 catalog_manager.cc:5484] T 36b48c566dcb445e97af8caaf6d4fe16 P 3f3cf1cac8e24dcc972a6d194f1903ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3cf1cac8e24dcc972a6d194f1903ab (127.10.145.1). New cstate: current_term: 1 leader_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.378135 25234 tablet_bootstrap.cc:654] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3: Neither blocks nor log segments found. Creating new log.
W20240503 14:15:06.380048 24900 tablet.cc:2369] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:06.381724 24579 catalog_manager.cc:5484] T 2afd5a389b1d4639b05a2b04baec64d1 P ce339dd856e343b19543edbb616974f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce339dd856e343b19543edbb616974f3 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ce339dd856e343b19543edbb616974f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.382392 25234 tablet_bootstrap.cc:492] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3: No bootstrap required, opened a new log
I20240503 14:15:06.382560 25234 ts_tablet_manager.cc:1367] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:15:06.383262 25234 raft_consensus.cc:348] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.383440 25234 raft_consensus.cc:374] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:06.383544 25234 raft_consensus.cc:729] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce339dd856e343b19543edbb616974f3, State: Initialized, Role: FOLLOWER
I20240503 14:15:06.383797 25234 consensus_queue.cc:260] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.383991 25234 raft_consensus.cc:388] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:06.384089 25234 raft_consensus.cc:482] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:06.384179 25234 raft_consensus.cc:3037] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:06.386813 25234 raft_consensus.cc:504] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.387082 25234 leader_election.cc:302] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce339dd856e343b19543edbb616974f3; no voters:
I20240503 14:15:06.387320 25234 leader_election.cc:288] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:06.387535 25238 raft_consensus.cc:2781] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:06.387735 25238 raft_consensus.cc:686] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [term 1 LEADER]: Becoming Leader. State: Replica: ce339dd856e343b19543edbb616974f3, State: Running, Role: LEADER
I20240503 14:15:06.387842 25234 ts_tablet_manager.cc:1398] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 14:15:06.387976 25238 consensus_queue.cc:237] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.390048 24579 catalog_manager.cc:5484] T 5b9242d41b414c6eb03c3fd3c43bad9b P ce339dd856e343b19543edbb616974f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce339dd856e343b19543edbb616974f3 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ce339dd856e343b19543edbb616974f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.921615 24581 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58294:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:06.930788 24702 tablet_service.cc:1467] Processing CreateTablet for tablet 10a36b686a5e43d983624018b0b9a7ff (DEFAULT_TABLE table=rebalance_test_table_2 [id=557eaded55a0474cac6a690e9f03aec3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:06.930778 24833 tablet_service.cc:1467] Processing CreateTablet for tablet d9e4096cfb204430bab06a674c885734 (DEFAULT_TABLE table=rebalance_test_table_2 [id=557eaded55a0474cac6a690e9f03aec3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:06.930961 24703 tablet_service.cc:1467] Processing CreateTablet for tablet 71ac2ad0fd5a4a10b3de6a488b3ed8a5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=557eaded55a0474cac6a690e9f03aec3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:06.931252 24702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a36b686a5e43d983624018b0b9a7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:06.931252 24833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e4096cfb204430bab06a674c885734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:06.931483 24703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71ac2ad0fd5a4a10b3de6a488b3ed8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:06.937093 25272 tablet_bootstrap.cc:492] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab: Bootstrap starting.
I20240503 14:15:06.938082 25273 tablet_bootstrap.cc:492] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3: Bootstrap starting.
I20240503 14:15:06.938832 25272 tablet_bootstrap.cc:654] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:06.939585 25273 tablet_bootstrap.cc:654] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:06.940557 25272 tablet_bootstrap.cc:492] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab: No bootstrap required, opened a new log
I20240503 14:15:06.940723 25272 ts_tablet_manager.cc:1367] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:06.941223 25273 tablet_bootstrap.cc:492] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3: No bootstrap required, opened a new log
I20240503 14:15:06.941391 25273 ts_tablet_manager.cc:1367] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:06.941378 25272 raft_consensus.cc:348] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.941540 25272 raft_consensus.cc:374] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:06.941634 25272 raft_consensus.cc:729] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Initialized, Role: FOLLOWER
I20240503 14:15:06.941872 25272 consensus_queue.cc:260] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.942061 25272 raft_consensus.cc:388] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:06.942144 25272 raft_consensus.cc:482] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:06.942250 25272 raft_consensus.cc:3037] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:06.942391 25273 raft_consensus.cc:348] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.942538 25273 raft_consensus.cc:374] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:06.942626 25273 raft_consensus.cc:729] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce339dd856e343b19543edbb616974f3, State: Initialized, Role: FOLLOWER
I20240503 14:15:06.942885 25273 consensus_queue.cc:260] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.943070 25273 raft_consensus.cc:388] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:06.943147 25273 raft_consensus.cc:482] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:06.943248 25273 raft_consensus.cc:3037] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:06.944291 25272 raft_consensus.cc:504] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.944509 25272 leader_election.cc:302] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3cf1cac8e24dcc972a6d194f1903ab; no voters:
I20240503 14:15:06.944938 25272 leader_election.cc:288] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:06.945083 25276 raft_consensus.cc:2781] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:06.945077 25273 raft_consensus.cc:504] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.945420 25273 leader_election.cc:302] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce339dd856e343b19543edbb616974f3; no voters:
I20240503 14:15:06.945734 25238 raft_consensus.cc:2781] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:06.945752 25272 ts_tablet_manager.cc:1398] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 14:15:06.946070 25272 tablet_bootstrap.cc:492] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab: Bootstrap starting.
I20240503 14:15:06.948249 25272 tablet_bootstrap.cc:654] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:06.951709 25276 raft_consensus.cc:686] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 LEADER]: Becoming Leader. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Running, Role: LEADER
I20240503 14:15:06.951754 25273 leader_election.cc:288] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:06.952149 25276 consensus_queue.cc:237] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.955518 25273 ts_tablet_manager.cc:1398] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240503 14:15:06.955404 24581 catalog_manager.cc:5484] T 71ac2ad0fd5a4a10b3de6a488b3ed8a5 P 3f3cf1cac8e24dcc972a6d194f1903ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3cf1cac8e24dcc972a6d194f1903ab (127.10.145.1). New cstate: current_term: 1 leader_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.955785 25238 raft_consensus.cc:686] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [term 1 LEADER]: Becoming Leader. State: Replica: ce339dd856e343b19543edbb616974f3, State: Running, Role: LEADER
I20240503 14:15:06.956130 25238 consensus_queue.cc:237] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:06.956827 25272 tablet_bootstrap.cc:492] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab: No bootstrap required, opened a new log
I20240503 14:15:06.957011 25272 ts_tablet_manager.cc:1367] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240503 14:15:06.959012 25272 raft_consensus.cc:348] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.959293 25272 raft_consensus.cc:374] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:06.959388 25272 raft_consensus.cc:729] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Initialized, Role: FOLLOWER
I20240503 14:15:06.959681 25272 consensus_queue.cc:260] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.959887 25272 raft_consensus.cc:388] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:06.960005 25272 raft_consensus.cc:482] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:06.960134 25272 raft_consensus.cc:3037] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:06.962744 25272 raft_consensus.cc:504] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.962986 25272 leader_election.cc:302] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3cf1cac8e24dcc972a6d194f1903ab; no voters:
I20240503 14:15:06.962795 24579 catalog_manager.cc:5484] T d9e4096cfb204430bab06a674c885734 P ce339dd856e343b19543edbb616974f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce339dd856e343b19543edbb616974f3 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ce339dd856e343b19543edbb616974f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:06.963177 25272 leader_election.cc:288] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:06.963235 25276 raft_consensus.cc:2781] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:06.963551 25276 raft_consensus.cc:686] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 LEADER]: Becoming Leader. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Running, Role: LEADER
I20240503 14:15:06.963712 25272 ts_tablet_manager.cc:1398] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240503 14:15:06.964530 25276 consensus_queue.cc:237] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:06.967605 24579 catalog_manager.cc:5484] T 10a36b686a5e43d983624018b0b9a7ff P 3f3cf1cac8e24dcc972a6d194f1903ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3cf1cac8e24dcc972a6d194f1903ab (127.10.145.1). New cstate: current_term: 1 leader_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:07.489638 24579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58294:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:07.499123 24833 tablet_service.cc:1467] Processing CreateTablet for tablet 691e8705a5ab494c86a3d9000b5866cc (DEFAULT_TABLE table=rebalance_test_table_3 [id=9ebd0c326d2e45868279bbc62edb6c52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:07.499123 24834 tablet_service.cc:1467] Processing CreateTablet for tablet 64a8bba3648d4a649577177a188c8a25 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9ebd0c326d2e45868279bbc62edb6c52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:07.499665 24833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691e8705a5ab494c86a3d9000b5866cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:07.499634 24702 tablet_service.cc:1467] Processing CreateTablet for tablet 29ce0ff2bde64a67a638ba24a9417683 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9ebd0c326d2e45868279bbc62edb6c52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:07.499974 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a8bba3648d4a649577177a188c8a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:07.500113 24702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29ce0ff2bde64a67a638ba24a9417683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:07.506237 25310 tablet_bootstrap.cc:492] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab: Bootstrap starting.
I20240503 14:15:07.506484 25311 tablet_bootstrap.cc:492] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3: Bootstrap starting.
I20240503 14:15:07.507666 25310 tablet_bootstrap.cc:654] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:07.507915 25311 tablet_bootstrap.cc:654] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:07.509703 25310 tablet_bootstrap.cc:492] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab: No bootstrap required, opened a new log
I20240503 14:15:07.509846 25310 ts_tablet_manager.cc:1367] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:07.510468 25310 raft_consensus.cc:348] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:07.510625 25310 raft_consensus.cc:374] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:07.510713 25310 raft_consensus.cc:729] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Initialized, Role: FOLLOWER
I20240503 14:15:07.510941 25310 consensus_queue.cc:260] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:07.511104 25310 raft_consensus.cc:388] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:07.511180 25310 raft_consensus.cc:482] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:07.511286 25310 raft_consensus.cc:3037] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:07.513468 25310 raft_consensus.cc:504] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:07.513721 25310 leader_election.cc:302] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3cf1cac8e24dcc972a6d194f1903ab; no voters:
I20240503 14:15:07.513968 25310 leader_election.cc:288] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:07.514094 25276 raft_consensus.cc:2781] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:07.514942 25310 ts_tablet_manager.cc:1398] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240503 14:15:07.515657 25276 raft_consensus.cc:686] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [term 1 LEADER]: Becoming Leader. State: Replica: 3f3cf1cac8e24dcc972a6d194f1903ab, State: Running, Role: LEADER
I20240503 14:15:07.515942 25276 consensus_queue.cc:237] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } }
I20240503 14:15:07.516225 25311 tablet_bootstrap.cc:492] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3: No bootstrap required, opened a new log
I20240503 14:15:07.516395 25311 ts_tablet_manager.cc:1367] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240503 14:15:07.517155 25311 raft_consensus.cc:348] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:07.517341 25311 raft_consensus.cc:374] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:07.517469 25311 raft_consensus.cc:729] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce339dd856e343b19543edbb616974f3, State: Initialized, Role: FOLLOWER
I20240503 14:15:07.517726 25311 consensus_queue.cc:260] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:07.518028 25311 raft_consensus.cc:388] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:07.518121 25311 raft_consensus.cc:482] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:07.518234 25311 raft_consensus.cc:3037] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:07.518810 24579 catalog_manager.cc:5484] T 29ce0ff2bde64a67a638ba24a9417683 P 3f3cf1cac8e24dcc972a6d194f1903ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3cf1cac8e24dcc972a6d194f1903ab (127.10.145.1). New cstate: current_term: 1 leader_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3cf1cac8e24dcc972a6d194f1903ab" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 37737 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:07.520628 25311 raft_consensus.cc:504] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:07.520897 25311 leader_election.cc:302] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce339dd856e343b19543edbb616974f3; no voters:
I20240503 14:15:07.521399 25311 leader_election.cc:288] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:07.522156 25317 raft_consensus.cc:2781] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:07.522181 25311 ts_tablet_manager.cc:1398] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240503 14:15:07.522341 25317 raft_consensus.cc:686] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [term 1 LEADER]: Becoming Leader. State: Replica: ce339dd856e343b19543edbb616974f3, State: Running, Role: LEADER
I20240503 14:15:07.522503 25311 tablet_bootstrap.cc:492] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3: Bootstrap starting.
I20240503 14:15:07.522636 25317 consensus_queue.cc:237] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:07.524363 25311 tablet_bootstrap.cc:654] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:07.525161 24581 catalog_manager.cc:5484] T 691e8705a5ab494c86a3d9000b5866cc P ce339dd856e343b19543edbb616974f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce339dd856e343b19543edbb616974f3 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ce339dd856e343b19543edbb616974f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:07.530405 25311 tablet_bootstrap.cc:492] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3: No bootstrap required, opened a new log
I20240503 14:15:07.530594 25311 ts_tablet_manager.cc:1367] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240503 14:15:07.531369 25311 raft_consensus.cc:348] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:07.531556 25311 raft_consensus.cc:374] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:07.531630 25311 raft_consensus.cc:729] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce339dd856e343b19543edbb616974f3, State: Initialized, Role: FOLLOWER
I20240503 14:15:07.531847 25311 consensus_queue.cc:260] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:07.531988 25311 raft_consensus.cc:388] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:07.532097 25311 raft_consensus.cc:482] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:07.532189 25311 raft_consensus.cc:3037] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:07.534229 25311 raft_consensus.cc:504] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:07.534458 25311 leader_election.cc:302] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce339dd856e343b19543edbb616974f3; no voters:
I20240503 14:15:07.534654 25311 leader_election.cc:288] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:07.534706 25317 raft_consensus.cc:2781] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:07.535166 25311 ts_tablet_manager.cc:1398] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:07.540360 25317 raft_consensus.cc:686] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [term 1 LEADER]: Becoming Leader. State: Replica: ce339dd856e343b19543edbb616974f3, State: Running, Role: LEADER
I20240503 14:15:07.540645 25317 consensus_queue.cc:237] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } }
I20240503 14:15:07.543242 24581 catalog_manager.cc:5484] T 64a8bba3648d4a649577177a188c8a25 P ce339dd856e343b19543edbb616974f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce339dd856e343b19543edbb616974f3 (127.10.145.2). New cstate: current_term: 1 leader_uuid: "ce339dd856e343b19543edbb616974f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce339dd856e343b19543edbb616974f3" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 45183 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:08.940062 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.3:46591
--local_ip_for_outbound_sockets=127.10.145.3
--tserver_master_addrs=127.10.145.62:42931
--webserver_port=44673
--webserver_interface=127.10.145.3
--builtin_ntp_servers=127.10.145.20:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:15:09.299264 25346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:09.299585 25346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:09.299703 25346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:09.307413 25346 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:09.307566 25346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:09.307731 25346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.3
I20240503 14:15:09.317885 25346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.3:46591
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.145.3
--webserver_port=44673
--tserver_master_addrs=127.10.145.62:42931
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.3
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:09.319337 25346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:09.321359 25346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:09.334417 25351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:09.334794 25352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:09.334887 25354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:09.335292 25346 server_base.cc:1034] running on GCE node
I20240503 14:15:09.336020 25346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:09.337018 25346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:09.338212 25346 hybrid_clock.cc:648] HybridClock initialized: now 1714745709338132 us; error 77 us; skew 500 ppm
I20240503 14:15:09.338544 25346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:09.348102 25346 webserver.cc:460] Webserver started at http://127.10.145.3:44673/ using document root <none> and password file <none>
I20240503 14:15:09.348979 25346 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:09.349136 25346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:09.354969 25346 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20240503 14:15:09.358265 25360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:09.359781 25346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:09.359964 25346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "35d33222882b409cbda9c6b44847500d"
format_stamp: "Formatted at 2024-05-03 14:15:01 on dist-test-slave-v549"
I20240503 14:15:09.360894 25346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:09.384023 25346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:09.384967 25346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:09.385273 25346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:09.386350 25346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:09.388535 25367 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 14:15:09.394555 25346 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 14:15:09.394694 25346 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20240503 14:15:09.394845 25346 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 14:15:09.398068 25346 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 14:15:09.398170 25346 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:09.398602 25367 tablet_bootstrap.cc:492] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Bootstrap starting.
I20240503 14:15:09.420361 25367 log.cc:824] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:09.430158 25367 tablet_bootstrap.cc:492] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:09.430799 25367 tablet_bootstrap.cc:492] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Bootstrap complete.
I20240503 14:15:09.432327 25367 ts_tablet_manager.cc:1367] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.013s
I20240503 14:15:09.443221 25367 raft_consensus.cc:348] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d33222882b409cbda9c6b44847500d" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46591 } }
I20240503 14:15:09.444479 25367 raft_consensus.cc:729] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d33222882b409cbda9c6b44847500d, State: Initialized, Role: FOLLOWER
I20240503 14:15:09.445335 25367 consensus_queue.cc:260] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d33222882b409cbda9c6b44847500d" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46591 } }
I20240503 14:15:09.445598 25367 raft_consensus.cc:388] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:09.445785 25367 raft_consensus.cc:482] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:09.446095 25367 raft_consensus.cc:3037] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:09.450469 25367 raft_consensus.cc:504] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d33222882b409cbda9c6b44847500d" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46591 } }
I20240503 14:15:09.451149 25367 leader_election.cc:302] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35d33222882b409cbda9c6b44847500d; no voters:
I20240503 14:15:09.451936 25367 leader_election.cc:288] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 14:15:09.452066 25468 raft_consensus.cc:2781] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:09.458413 25346 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.3:46591
I20240503 14:15:09.458456 25477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.3:46591 every 8 connection(s)
I20240503 14:15:09.460148 25346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:15:09.461050 25367 ts_tablet_manager.cc:1398] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d: Time spent starting tablet: real 0.029s user 0.021s sys 0.008s
I20240503 14:15:09.461870 25468 raft_consensus.cc:686] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [term 2 LEADER]: Becoming Leader. State: Replica: 35d33222882b409cbda9c6b44847500d, State: Running, Role: LEADER
I20240503 14:15:09.462424 25468 consensus_queue.cc:237] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d33222882b409cbda9c6b44847500d" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46591 } }
I20240503 14:15:09.463892 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 25346
I20240503 14:15:09.464824 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.4:39503
--local_ip_for_outbound_sockets=127.10.145.4
--tserver_master_addrs=127.10.145.62:42931
--webserver_port=46357
--webserver_interface=127.10.145.4
--builtin_ntp_servers=127.10.145.20:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:09.489483 25478 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42931
I20240503 14:15:09.489827 25478 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:09.490751 25478 heartbeater.cc:511] Master 127.10.145.62:42931 requested a full tablet report, sending...
I20240503 14:15:09.493928 24581 ts_manager.cc:194] Re-registered known tserver with Master: 35d33222882b409cbda9c6b44847500d (127.10.145.3:46591)
I20240503 14:15:09.494474 24581 catalog_manager.cc:5484] T 59f53dd9e5bc4475b314a98c85f7a7eb P 35d33222882b409cbda9c6b44847500d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "35d33222882b409cbda9c6b44847500d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35d33222882b409cbda9c6b44847500d" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46591 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:09.499756 24581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.3:32923
I20240503 14:15:09.502875 25478 heartbeater.cc:503] Master 127.10.145.62:42931 was elected leader, sending a full tablet report...
W20240503 14:15:09.826967 25484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:09.827335 25484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:09.827452 25484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:09.835352 25484 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:09.835505 25484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:09.835662 25484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.4
I20240503 14:15:09.845422 25484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.4:39503
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.145.4
--webserver_port=46357
--tserver_master_addrs=127.10.145.62:42931
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.4
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:09.846969 25484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:09.848886 25484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:09.862064 25494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:09.862401 25496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:09.862879 25493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:09.863289 25484 server_base.cc:1034] running on GCE node
I20240503 14:15:09.864004 25484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:09.865104 25484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:09.866370 25484 hybrid_clock.cc:648] HybridClock initialized: now 1714745709866301 us; error 80 us; skew 500 ppm
I20240503 14:15:09.866747 25484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:09.869228 25484 webserver.cc:460] Webserver started at http://127.10.145.4:46357/ using document root <none> and password file <none>
I20240503 14:15:09.870131 25484 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:09.870275 25484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:09.876341 25484 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20240503 14:15:09.879513 25502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:09.881013 25484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240503 14:15:09.881197 25484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0709ec44591a468781515e7157d6b1e5"
format_stamp: "Formatted at 2024-05-03 14:15:01 on dist-test-slave-v549"
I20240503 14:15:09.882138 25484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:09.898165 25484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:09.899123 25484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:09.899417 25484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:09.900454 25484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:09.902295 25484 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:09.902433 25484 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20240503 14:15:09.902572 25484 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:09.902655 25484 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:09.933511 25484 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.4:39503
I20240503 14:15:09.933571 25614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.4:39503 every 8 connection(s)
I20240503 14:15:09.935024 25484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:15:09.940068 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 25484
I20240503 14:15:09.948081 25615 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42931
I20240503 14:15:09.948406 25615 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:09.949324 25615 heartbeater.cc:511] Master 127.10.145.62:42931 requested a full tablet report, sending...
I20240503 14:15:09.950892 24581 ts_manager.cc:194] Re-registered known tserver with Master: 0709ec44591a468781515e7157d6b1e5 (127.10.145.4:39503)
I20240503 14:15:09.951994 24581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.4:56625
I20240503 14:15:10.458271 25550 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:10.463513 24702 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:15:10.469588 24833 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:15:10.485765 25410 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:10.663260 25550 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:10.664309 24833 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:15:10.670286 24702 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:15:10.676789 25410 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:10.927596 25550 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:10.929996 25410 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:10.944923 24833 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:15:10.951149 24702 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:15:10.954681 25615 heartbeater.cc:503] Master 127.10.145.62:42931 was elected leader, sending a full tablet report...
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7c8e126a71a341c7ab77674a53ab2f7a | 127.10.145.62:42931 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.145.20:37517 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0709ec44591a468781515e7157d6b1e5 | 127.10.145.4:39503 | HEALTHY | <none> | 0 | 0
35d33222882b409cbda9c6b44847500d | 127.10.145.3:46591 | HEALTHY | <none> | 1 | 0
3f3cf1cac8e24dcc972a6d194f1903ab | 127.10.145.1:37737 | HEALTHY | <none> | 6 | 0
ce339dd856e343b19543edbb616974f3 | 127.10.145.2:45183 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.10.145.1 | experimental | 127.10.145.1:37737
local_ip_for_outbound_sockets | 127.10.145.2 | experimental | 127.10.145.2:45183
local_ip_for_outbound_sockets | 127.10.145.3 | experimental | 127.10.145.3:46591
local_ip_for_outbound_sockets | 127.10.145.4 | experimental | 127.10.145.4:39503
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.145.1:37737
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.145.2:45183
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.145.3:46591
server_dump_info_path | /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.10.145.4:39503
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.145.20:37517 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 6
Third Quartile | 6
Maximum | 6
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:15:10.994558 10820 log_verifier.cc:126] Checking tablet 10a36b686a5e43d983624018b0b9a7ff
I20240503 14:15:11.002677 10820 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 10a36b686a5e43d983624018b0b9a7ff
I20240503 14:15:11.002772 10820 log_verifier.cc:126] Checking tablet 29ce0ff2bde64a67a638ba24a9417683
I20240503 14:15:11.010166 10820 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 29ce0ff2bde64a67a638ba24a9417683
I20240503 14:15:11.010263 10820 log_verifier.cc:126] Checking tablet 2afd5a389b1d4639b05a2b04baec64d1
I20240503 14:15:11.017961 10820 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 2afd5a389b1d4639b05a2b04baec64d1
I20240503 14:15:11.018064 10820 log_verifier.cc:126] Checking tablet 36b48c566dcb445e97af8caaf6d4fe16
I20240503 14:15:11.025413 10820 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 36b48c566dcb445e97af8caaf6d4fe16
I20240503 14:15:11.025507 10820 log_verifier.cc:126] Checking tablet 3b60eae0eb9447c79a498d3d13427da3
I20240503 14:15:11.033179 10820 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 3b60eae0eb9447c79a498d3d13427da3
I20240503 14:15:11.033272 10820 log_verifier.cc:126] Checking tablet 59f53dd9e5bc4475b314a98c85f7a7eb
I20240503 14:15:11.040730 10820 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 59f53dd9e5bc4475b314a98c85f7a7eb
I20240503 14:15:11.040830 10820 log_verifier.cc:126] Checking tablet 5b9242d41b414c6eb03c3fd3c43bad9b
I20240503 14:15:11.048552 10820 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5b9242d41b414c6eb03c3fd3c43bad9b
I20240503 14:15:11.048650 10820 log_verifier.cc:126] Checking tablet 64a8bba3648d4a649577177a188c8a25
I20240503 14:15:11.056231 10820 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 64a8bba3648d4a649577177a188c8a25
I20240503 14:15:11.056335 10820 log_verifier.cc:126] Checking tablet 691e8705a5ab494c86a3d9000b5866cc
I20240503 14:15:11.064193 10820 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 691e8705a5ab494c86a3d9000b5866cc
I20240503 14:15:11.064297 10820 log_verifier.cc:126] Checking tablet 71ac2ad0fd5a4a10b3de6a488b3ed8a5
I20240503 14:15:11.071652 10820 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 71ac2ad0fd5a4a10b3de6a488b3ed8a5
I20240503 14:15:11.071746 10820 log_verifier.cc:126] Checking tablet b25a4c1300a148bfa4259093428dbd85
I20240503 14:15:11.079107 10820 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b25a4c1300a148bfa4259093428dbd85
I20240503 14:15:11.079206 10820 log_verifier.cc:126] Checking tablet d61d8fea94b748f1a62e9ade509782a2
I20240503 14:15:11.086771 10820 log_verifier.cc:177] Verified matching terms for 2 ops in tablet d61d8fea94b748f1a62e9ade509782a2
I20240503 14:15:11.086863 10820 log_verifier.cc:126] Checking tablet d9e4096cfb204430bab06a674c885734
I20240503 14:15:11.094254 10820 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d9e4096cfb204430bab06a674c885734
I20240503 14:15:11.095182 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 24641
I20240503 14:15:11.096577 24763 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:11.197101 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 24641
I20240503 14:15:11.214454 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 24772
I20240503 14:15:11.215548 24894 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:11.315372 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 24772
I20240503 14:15:11.332365 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 25346
I20240503 14:15:11.339107 25472 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:11.432971 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 25346
I20240503 14:15:11.448745 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 25484
I20240503 14:15:11.455377 25609 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:11.543594 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 25484
I20240503 14:15:11.564463 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 24550
I20240503 14:15:11.565541 24611 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:11.683830 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 24550
2024-05-03T14:15:11Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 (12703 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (12703 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1
I20240503 14:15:11.734930 10820 test_util.cc:268] Using random seed: 1493643383
I20240503 14:15:11.736897 10820 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:15:11.736963 10820 ts_itest-base.cc:113] --------------
I20240503 14:15:11.737005 10820 ts_itest-base.cc:114] 4 tablet servers
I20240503 14:15:11.737056 10820 ts_itest-base.cc:115] 2 replicas per TS
I20240503 14:15:11.737113 10820 ts_itest-base.cc:116] --------------
2024-05-03T14:15:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:15:11Z Disabled control of system clock
I20240503 14:15:11.787930 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:33795
--webserver_interface=127.10.145.62
--webserver_port=0
--builtin_ntp_servers=127.10.145.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.145.62:33795
--location_mapping_cmd=/tmp/dist-test-taskFZBpI6/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 14:15:12.141171 25729 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 14:15:12.141597 25729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:12.141705 25729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:12.141819 25729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:12.149467 25729 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:12.149603 25729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:15:12.149703 25729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:12.149756 25729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:15:12.149839 25729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:15:12.159402 25729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskFZBpI6/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--ipki_ca_key_size=768
--master_addresses=127.10.145.62:33795
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:33795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:12.160931 25729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:12.162931 25729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:12.174803 25735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:12.174958 25734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:12.175357 25737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:12.175774 25729 server_base.cc:1034] running on GCE node
I20240503 14:15:12.176630 25729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:12.178346 25729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:12.179579 25729 hybrid_clock.cc:648] HybridClock initialized: now 1714745712179512 us; error 53 us; skew 500 ppm
I20240503 14:15:12.179919 25729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:12.182464 25729 webserver.cc:460] Webserver started at http://127.10.145.62:38717/ using document root <none> and password file <none>
I20240503 14:15:12.183269 25729 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:12.183403 25729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:12.183725 25729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:12.186646 25729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6d6b298fbc7a43a7bdc3514cd111936c"
format_stamp: "Formatted at 2024-05-03 14:15:12 on dist-test-slave-v549"
I20240503 14:15:12.187547 25729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6d6b298fbc7a43a7bdc3514cd111936c"
format_stamp: "Formatted at 2024-05-03 14:15:12 on dist-test-slave-v549"
I20240503 14:15:12.193250 25729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240503 14:15:12.197136 25743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:12.198666 25729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 14:15:12.198848 25729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6d6b298fbc7a43a7bdc3514cd111936c"
format_stamp: "Formatted at 2024-05-03 14:15:12 on dist-test-slave-v549"
I20240503 14:15:12.199039 25729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:12.224337 25729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:12.225311 25729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:12.225626 25729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:12.243778 25729 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.62:33795
I20240503 14:15:12.243798 25794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.62:33795 every 8 connection(s)
I20240503 14:15:12.245273 25729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:15:12.248590 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 25729
I20240503 14:15:12.249622 25795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:12.259575 25795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c: Bootstrap starting.
I20240503 14:15:12.263386 25795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:12.264485 25795 log.cc:824] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:12.267530 25795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c: No bootstrap required, opened a new log
I20240503 14:15:12.274736 25795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b298fbc7a43a7bdc3514cd111936c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33795 } }
I20240503 14:15:12.275161 25795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:12.275272 25795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6b298fbc7a43a7bdc3514cd111936c, State: Initialized, Role: FOLLOWER
I20240503 14:15:12.275976 25795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b298fbc7a43a7bdc3514cd111936c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33795 } }
I20240503 14:15:12.276190 25795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:12.276319 25795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:12.276472 25795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:12.278659 25795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b298fbc7a43a7bdc3514cd111936c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33795 } }
I20240503 14:15:12.279209 25795 leader_election.cc:302] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d6b298fbc7a43a7bdc3514cd111936c; no voters:
I20240503 14:15:12.279917 25795 leader_election.cc:288] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:12.280041 25798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:12.281149 25798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [term 1 LEADER]: Becoming Leader. State: Replica: 6d6b298fbc7a43a7bdc3514cd111936c, State: Running, Role: LEADER
I20240503 14:15:12.281850 25798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b298fbc7a43a7bdc3514cd111936c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33795 } }
I20240503 14:15:12.282387 25795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:15:12.285403 25800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d6b298fbc7a43a7bdc3514cd111936c. Latest consensus state: current_term: 1 leader_uuid: "6d6b298fbc7a43a7bdc3514cd111936c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b298fbc7a43a7bdc3514cd111936c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33795 } } }
I20240503 14:15:12.285432 25799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d6b298fbc7a43a7bdc3514cd111936c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b298fbc7a43a7bdc3514cd111936c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 33795 } } }
I20240503 14:15:12.285780 25800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [sys.catalog]: This master's current role is: LEADER
I20240503 14:15:12.285801 25799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c [sys.catalog]: This master's current role is: LEADER
I20240503 14:15:12.290064 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:15:12.290091 25805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:15:12.296222 25805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:15:12.309454 25805 catalog_manager.cc:1347] Generated new cluster ID: f152f0d1ba014aae8a8d2cdd9302e97f
I20240503 14:15:12.309603 25805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:15:12.321879 25805 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:15:12.323352 25805 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:15:12.331809 25805 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6d6b298fbc7a43a7bdc3514cd111936c: Generated new TSK 0
I20240503 14:15:12.332782 25805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:15:12.347640 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.1:0
--local_ip_for_outbound_sockets=127.10.145.1
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33795
--builtin_ntp_servers=127.10.145.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:15:12.741065 25819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:12.741431 25819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:12.741568 25819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:12.749214 25819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:12.749387 25819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:12.749555 25819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.1
I20240503 14:15:12.758846 25819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.1
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:12.760291 25819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:12.762250 25819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:12.774698 25824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:12.774654 25827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:12.775226 25825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:12.775527 25819 server_base.cc:1034] running on GCE node
I20240503 14:15:12.776110 25819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:12.777088 25819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:12.778291 25819 hybrid_clock.cc:648] HybridClock initialized: now 1714745712778235 us; error 54 us; skew 500 ppm
I20240503 14:15:12.778632 25819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:12.780956 25819 webserver.cc:460] Webserver started at http://127.10.145.1:44597/ using document root <none> and password file <none>
I20240503 14:15:12.781781 25819 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:12.781895 25819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:12.782212 25819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:12.785147 25819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6dd1204c91174803920e3a83d494a1fe"
format_stamp: "Formatted at 2024-05-03 14:15:12 on dist-test-slave-v549"
I20240503 14:15:12.785991 25819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6dd1204c91174803920e3a83d494a1fe"
format_stamp: "Formatted at 2024-05-03 14:15:12 on dist-test-slave-v549"
I20240503 14:15:12.791374 25819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:15:12.794885 25833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:12.796348 25819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:15:12.796517 25819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6dd1204c91174803920e3a83d494a1fe"
format_stamp: "Formatted at 2024-05-03 14:15:12 on dist-test-slave-v549"
I20240503 14:15:12.796716 25819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:12.829190 25819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:12.830173 25819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:12.830463 25819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:12.831506 25819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:12.833082 25819 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:12.833182 25819 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:12.833314 25819 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:12.833396 25819 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 14:15:12.865648 25819 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.1:45305
I20240503 14:15:12.865700 25945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.1:45305 every 8 connection(s)
I20240503 14:15:12.867031 25819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:15:12.868039 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 25819
I20240503 14:15:12.868299 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:15:12.872071 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.2:0
--local_ip_for_outbound_sockets=127.10.145.2
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33795
--builtin_ntp_servers=127.10.145.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:12.881253 25946 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33795
I20240503 14:15:12.881668 25946 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:12.882786 25946 heartbeater.cc:511] Master 127.10.145.62:33795 requested a full tablet report, sending...
I20240503 14:15:12.930153 25760 ts_manager.cc:194] Registered new tserver with Master: 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:12.932018 25760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.1:36959
W20240503 14:15:13.234151 25950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:13.234519 25950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:13.234658 25950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:13.242475 25950 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:13.242635 25950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:13.242794 25950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.2
I20240503 14:15:13.252367 25950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.2
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:13.253940 25950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:13.255870 25950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:13.268532 25959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:13.268990 25956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:13.269340 25957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:13.269755 25950 server_base.cc:1034] running on GCE node
I20240503 14:15:13.270432 25950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:13.271502 25950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:13.272750 25950 hybrid_clock.cc:648] HybridClock initialized: now 1714745713272686 us; error 52 us; skew 500 ppm
I20240503 14:15:13.273169 25950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:13.275923 25950 webserver.cc:460] Webserver started at http://127.10.145.2:44683/ using document root <none> and password file <none>
I20240503 14:15:13.276763 25950 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:13.276883 25950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:13.277228 25950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:13.280277 25950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a680f58704cf4c979e241a706d51b972"
format_stamp: "Formatted at 2024-05-03 14:15:13 on dist-test-slave-v549"
I20240503 14:15:13.281150 25950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a680f58704cf4c979e241a706d51b972"
format_stamp: "Formatted at 2024-05-03 14:15:13 on dist-test-slave-v549"
I20240503 14:15:13.287083 25950 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.002s
I20240503 14:15:13.291059 25965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:13.292623 25950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:15:13.292807 25950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a680f58704cf4c979e241a706d51b972"
format_stamp: "Formatted at 2024-05-03 14:15:13 on dist-test-slave-v549"
I20240503 14:15:13.293025 25950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:13.311704 25950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:13.312636 25950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:13.312918 25950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:13.314041 25950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:13.315728 25950 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:13.315827 25950 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:13.315945 25950 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:13.316015 25950 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:13.346534 25950 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.2:35219
I20240503 14:15:13.346593 26077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.2:35219 every 8 connection(s)
I20240503 14:15:13.347944 25950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:15:13.349499 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 25950
I20240503 14:15:13.349660 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:15:13.353368 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.3:0
--local_ip_for_outbound_sockets=127.10.145.3
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33795
--builtin_ntp_servers=127.10.145.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:13.361593 26078 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33795
I20240503 14:15:13.361920 26078 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:13.362789 26078 heartbeater.cc:511] Master 127.10.145.62:33795 requested a full tablet report, sending...
I20240503 14:15:13.410368 25760 ts_manager.cc:194] Registered new tserver with Master: a680f58704cf4c979e241a706d51b972 (127.10.145.2:35219)
I20240503 14:15:13.411707 25760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.2:51711
W20240503 14:15:13.707429 26081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:13.707752 26081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:13.707839 26081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:13.715615 26081 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:13.715749 26081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:13.715919 26081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.3
I20240503 14:15:13.725111 26081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.3
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:13.726764 26081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:13.728605 26081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:13.741175 26087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:13.741461 26088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:13.741781 26090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:13.742165 26081 server_base.cc:1034] running on GCE node
I20240503 14:15:13.742754 26081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:13.743721 26081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:13.744926 26081 hybrid_clock.cc:648] HybridClock initialized: now 1714745713744867 us; error 61 us; skew 500 ppm
I20240503 14:15:13.745252 26081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:13.747789 26081 webserver.cc:460] Webserver started at http://127.10.145.3:41577/ using document root <none> and password file <none>
I20240503 14:15:13.748692 26081 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:13.748788 26081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:13.749068 26081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:13.751832 26081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "171289531539427c84649caa8aa7f86e"
format_stamp: "Formatted at 2024-05-03 14:15:13 on dist-test-slave-v549"
I20240503 14:15:13.752612 26081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "171289531539427c84649caa8aa7f86e"
format_stamp: "Formatted at 2024-05-03 14:15:13 on dist-test-slave-v549"
I20240503 14:15:13.758136 26081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 14:15:13.761693 26096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:13.763144 26081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:15:13.763311 26081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "171289531539427c84649caa8aa7f86e"
format_stamp: "Formatted at 2024-05-03 14:15:13 on dist-test-slave-v549"
I20240503 14:15:13.763531 26081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:13.786266 26081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:13.787297 26081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:13.787714 26081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:13.788842 26081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:13.790627 26081 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:13.790735 26081 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:13.790843 26081 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:13.790913 26081 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:13.820976 26081 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.3:46613
I20240503 14:15:13.821050 26208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.3:46613 every 8 connection(s)
I20240503 14:15:13.822414 26081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:15:13.828342 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 26081
I20240503 14:15:13.828575 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:15:13.832342 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.4:0
--local_ip_for_outbound_sockets=127.10.145.4
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33795
--builtin_ntp_servers=127.10.145.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:13.846413 26209 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33795
I20240503 14:15:13.846922 26209 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:13.848042 26209 heartbeater.cc:511] Master 127.10.145.62:33795 requested a full tablet report, sending...
I20240503 14:15:13.895665 25760 ts_manager.cc:194] Registered new tserver with Master: 171289531539427c84649caa8aa7f86e (127.10.145.3:46613)
I20240503 14:15:13.896685 25760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.3:40523
I20240503 14:15:13.935320 25946 heartbeater.cc:503] Master 127.10.145.62:33795 was elected leader, sending a full tablet report...
W20240503 14:15:14.186177 26212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:14.186564 26212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:14.186676 26212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:14.194288 26212 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:14.194427 26212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:14.194555 26212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.4
I20240503 14:15:14.204083 26212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:33795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.4
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:14.205554 26212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:14.207399 26212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:14.220050 26219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:14.220357 26221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:14.220727 26218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:14.221143 26212 server_base.cc:1034] running on GCE node
I20240503 14:15:14.221808 26212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:14.222846 26212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:14.224048 26212 hybrid_clock.cc:648] HybridClock initialized: now 1714745714223990 us; error 51 us; skew 500 ppm
I20240503 14:15:14.224380 26212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:14.226861 26212 webserver.cc:460] Webserver started at http://127.10.145.4:44117/ using document root <none> and password file <none>
I20240503 14:15:14.227677 26212 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:14.227795 26212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:14.228114 26212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:14.230991 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0d5deb936bb744b392e3a28be25aa55c"
format_stamp: "Formatted at 2024-05-03 14:15:14 on dist-test-slave-v549"
I20240503 14:15:14.231812 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0d5deb936bb744b392e3a28be25aa55c"
format_stamp: "Formatted at 2024-05-03 14:15:14 on dist-test-slave-v549"
I20240503 14:15:14.237464 26212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:15:14.241482 26227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:14.243048 26212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:14.243252 26212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0d5deb936bb744b392e3a28be25aa55c"
format_stamp: "Formatted at 2024-05-03 14:15:14 on dist-test-slave-v549"
I20240503 14:15:14.243551 26212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:14.265949 26212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:14.266855 26212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:14.267115 26212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:14.268139 26212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:14.269869 26212 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:14.269968 26212 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:14.270049 26212 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:14.270102 26212 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:14.300565 26212 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.4:37471
I20240503 14:15:14.300612 26339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.4:37471 every 8 connection(s)
I20240503 14:15:14.302129 26212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:15:14.311906 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 26212
I20240503 14:15:14.312070 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:15:14.315869 26340 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33795
I20240503 14:15:14.316217 26340 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:14.317091 26340 heartbeater.cc:511] Master 127.10.145.62:33795 requested a full tablet report, sending...
I20240503 14:15:14.361255 25760 ts_manager.cc:194] Registered new tserver with Master: 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:14.362315 25760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.4:54899
I20240503 14:15:14.367202 10820 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 14:15:14.388324 10820 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 14:15:14.388523 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 25950
I20240503 14:15:14.397116 26073 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:14.478785 26078 heartbeater.cc:503] Master 127.10.145.62:33795 was elected leader, sending a full tablet report...
I20240503 14:15:14.480280 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 25950
W20240503 14:15:14.495507 25747 connection.cc:535] server connection from 127.10.145.2:51711 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:14.495766 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 26081
I20240503 14:15:14.504400 26204 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:14.586105 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 26081
I20240503 14:15:15.365573 26340 heartbeater.cc:503] Master 127.10.145.62:33795 was elected leader, sending a full tablet report...
I20240503 14:15:18.352741 25760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38548:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:18.359678 25760 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 14:15:18.391180 26273 tablet_service.cc:1467] Processing CreateTablet for tablet 209eb1a4b190422ea47b12f96755f064 (DEFAULT_TABLE table=rebalance_test_table_0 [id=938e33bb6bfb4575bf63a3643c9ee885]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:18.391114 25881 tablet_service.cc:1467] Processing CreateTablet for tablet b70ada8e0ac24fd5a7a4d7760051d197 (DEFAULT_TABLE table=rebalance_test_table_0 [id=938e33bb6bfb4575bf63a3643c9ee885]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:18.391208 26274 tablet_service.cc:1467] Processing CreateTablet for tablet 44de804572554d56becf3387e1031e1b (DEFAULT_TABLE table=rebalance_test_table_0 [id=938e33bb6bfb4575bf63a3643c9ee885]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:18.392819 26274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44de804572554d56becf3387e1031e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:18.392802 25881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70ada8e0ac24fd5a7a4d7760051d197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:18.391127 26275 tablet_service.cc:1467] Processing CreateTablet for tablet b70ada8e0ac24fd5a7a4d7760051d197 (DEFAULT_TABLE table=rebalance_test_table_0 [id=938e33bb6bfb4575bf63a3643c9ee885]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:18.393350 26273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209eb1a4b190422ea47b12f96755f064. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:18.393553 25879 tablet_service.cc:1467] Processing CreateTablet for tablet 209eb1a4b190422ea47b12f96755f064 (DEFAULT_TABLE table=rebalance_test_table_0 [id=938e33bb6bfb4575bf63a3643c9ee885]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:18.393754 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70ada8e0ac24fd5a7a4d7760051d197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:18.394061 25879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209eb1a4b190422ea47b12f96755f064. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:18.397652 25880 tablet_service.cc:1467] Processing CreateTablet for tablet 44de804572554d56becf3387e1031e1b (DEFAULT_TABLE table=rebalance_test_table_0 [id=938e33bb6bfb4575bf63a3643c9ee885]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:18.398160 25880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44de804572554d56becf3387e1031e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:18.413126 26362 tablet_bootstrap.cc:492] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c: Bootstrap starting.
I20240503 14:15:18.416817 26362 tablet_bootstrap.cc:654] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:18.417673 26363 tablet_bootstrap.cc:492] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: Bootstrap starting.
I20240503 14:15:18.418463 26362 log.cc:824] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:18.421320 26363 tablet_bootstrap.cc:654] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:18.421715 26362 tablet_bootstrap.cc:492] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c: No bootstrap required, opened a new log
I20240503 14:15:18.422019 26362 ts_tablet_manager.cc:1367] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 14:15:18.422626 26363 log.cc:824] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:18.425536 26363 tablet_bootstrap.cc:492] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: No bootstrap required, opened a new log
I20240503 14:15:18.425807 26363 ts_tablet_manager.cc:1367] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20240503 14:15:18.430716 26362 raft_consensus.cc:348] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.431190 26362 raft_consensus.cc:374] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:18.431298 26362 raft_consensus.cc:729] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.432041 26362 consensus_queue.cc:260] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.433919 26362 ts_tablet_manager.cc:1398] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20240503 14:15:18.434427 26362 tablet_bootstrap.cc:492] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c: Bootstrap starting.
I20240503 14:15:18.434283 26363 raft_consensus.cc:348] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.434749 26363 raft_consensus.cc:374] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:18.434912 26363 raft_consensus.cc:729] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.435644 26363 consensus_queue.cc:260] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.436244 26362 tablet_bootstrap.cc:654] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:18.439393 26363 ts_tablet_manager.cc:1398] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20240503 14:15:18.439863 26363 tablet_bootstrap.cc:492] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe: Bootstrap starting.
I20240503 14:15:18.441488 26362 tablet_bootstrap.cc:492] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c: No bootstrap required, opened a new log
I20240503 14:15:18.441529 26363 tablet_bootstrap.cc:654] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:18.441644 26362 ts_tablet_manager.cc:1367] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240503 14:15:18.442518 26362 raft_consensus.cc:348] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.442700 26362 raft_consensus.cc:374] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:18.442797 26362 raft_consensus.cc:729] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.443055 26362 consensus_queue.cc:260] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.443464 26363 tablet_bootstrap.cc:492] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe: No bootstrap required, opened a new log
I20240503 14:15:18.443567 26362 ts_tablet_manager.cc:1398] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:18.443647 26363 ts_tablet_manager.cc:1367] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:15:18.443943 26362 tablet_bootstrap.cc:492] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c: Bootstrap starting.
I20240503 14:15:18.444655 26363 raft_consensus.cc:348] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:18.444839 26363 raft_consensus.cc:374] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:18.444977 26363 raft_consensus.cc:729] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.445268 26363 consensus_queue.cc:260] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:18.445698 26362 tablet_bootstrap.cc:654] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:18.445791 26363 ts_tablet_manager.cc:1398] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:18.446056 26363 tablet_bootstrap.cc:492] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe: Bootstrap starting.
I20240503 14:15:18.447103 26367 raft_consensus.cc:482] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:18.447383 26367 raft_consensus.cc:504] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.447716 26363 tablet_bootstrap.cc:654] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:18.449885 26367 leader_election.cc:288] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:18.456475 26362 tablet_bootstrap.cc:492] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c: No bootstrap required, opened a new log
I20240503 14:15:18.456693 26362 ts_tablet_manager.cc:1367] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240503 14:15:18.456889 26363 tablet_bootstrap.cc:492] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe: No bootstrap required, opened a new log
I20240503 14:15:18.457063 26363 ts_tablet_manager.cc:1367] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240503 14:15:18.457574 26362 raft_consensus.cc:348] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:18.457839 26362 raft_consensus.cc:374] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:18.457805 26363 raft_consensus.cc:348] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.457934 26362 raft_consensus.cc:729] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.458017 26363 raft_consensus.cc:374] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:18.458086 26363 raft_consensus.cc:729] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.458223 26362 consensus_queue.cc:260] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:18.458335 26363 consensus_queue.cc:260] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.458797 26363 ts_tablet_manager.cc:1398] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:18.458812 26362 ts_tablet_manager.cc:1398] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:18.463747 26295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209eb1a4b190422ea47b12f96755f064" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5deb936bb744b392e3a28be25aa55c" is_pre_election: true
I20240503 14:15:18.464211 26295 raft_consensus.cc:2454] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 0.
I20240503 14:15:18.464742 25835 leader_election.cc:302] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:18.465116 26367 raft_consensus.cc:2781] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:18.465304 26367 raft_consensus.cc:482] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:18.465444 26367 raft_consensus.cc:3037] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:18.467573 26367 raft_consensus.cc:504] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.468134 26367 leader_election.cc:288] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 election: Requested vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:18.468632 26295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209eb1a4b190422ea47b12f96755f064" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5deb936bb744b392e3a28be25aa55c"
I20240503 14:15:18.468847 26295 raft_consensus.cc:3037] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:18.471029 26295 raft_consensus.cc:2454] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 1.
I20240503 14:15:18.471414 25835 leader_election.cc:302] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:18.471714 26367 raft_consensus.cc:2781] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:18.472142 26367 raft_consensus.cc:686] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Becoming Leader. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Running, Role: LEADER
I20240503 14:15:18.472812 26367 consensus_queue.cc:237] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.477596 25760 catalog_manager.cc:5484] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dd1204c91174803920e3a83d494a1fe (127.10.145.1). New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:18.510397 26367 raft_consensus.cc:482] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:18.510653 26367 raft_consensus.cc:504] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.511205 26367 leader_election.cc:288] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:18.511919 26295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70ada8e0ac24fd5a7a4d7760051d197" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5deb936bb744b392e3a28be25aa55c" is_pre_election: true
I20240503 14:15:18.512217 26295 raft_consensus.cc:2454] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 0.
I20240503 14:15:18.512645 25835 leader_election.cc:302] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:18.512990 26367 raft_consensus.cc:2781] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:18.513106 26367 raft_consensus.cc:482] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:18.513212 26367 raft_consensus.cc:3037] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:18.515179 26367 raft_consensus.cc:504] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.515677 26367 leader_election.cc:288] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 election: Requested vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:18.516271 26295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70ada8e0ac24fd5a7a4d7760051d197" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5deb936bb744b392e3a28be25aa55c"
I20240503 14:15:18.516485 26295 raft_consensus.cc:3037] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:18.518430 26295 raft_consensus.cc:2454] T b70ada8e0ac24fd5a7a4d7760051d197 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 1.
I20240503 14:15:18.518858 25835 leader_election.cc:302] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:18.519140 26367 raft_consensus.cc:2781] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:18.519297 26367 raft_consensus.cc:686] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Becoming Leader. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Running, Role: LEADER
I20240503 14:15:18.519556 26367 consensus_queue.cc:237] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:18.522224 25760 catalog_manager.cc:5484] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dd1204c91174803920e3a83d494a1fe (127.10.145.1). New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:18.555868 26367 raft_consensus.cc:482] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:18.556077 26367 raft_consensus.cc:504] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:18.556687 26367 leader_election.cc:288] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
W20240503 14:15:18.557252 26341 tablet.cc:2369] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:18.557446 26295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44de804572554d56becf3387e1031e1b" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5deb936bb744b392e3a28be25aa55c" is_pre_election: true
I20240503 14:15:18.557689 26295 raft_consensus.cc:2454] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 0.
I20240503 14:15:18.558149 25835 leader_election.cc:302] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:18.558440 26367 raft_consensus.cc:2781] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:18.558545 26367 raft_consensus.cc:482] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:18.558629 26367 raft_consensus.cc:3037] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:18.560480 26367 raft_consensus.cc:504] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:18.560945 26367 leader_election.cc:288] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 election: Requested vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:18.561578 26295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44de804572554d56becf3387e1031e1b" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5deb936bb744b392e3a28be25aa55c"
I20240503 14:15:18.561754 26295 raft_consensus.cc:3037] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:18.563614 26295 raft_consensus.cc:2454] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 1.
I20240503 14:15:18.564015 25835 leader_election.cc:302] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:18.564321 26367 raft_consensus.cc:2781] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:18.564455 26367 raft_consensus.cc:686] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Becoming Leader. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Running, Role: LEADER
I20240503 14:15:18.564747 26367 consensus_queue.cc:237] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:18.567322 25760 catalog_manager.cc:5484] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dd1204c91174803920e3a83d494a1fe (127.10.145.1). New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: UNKNOWN } } }
W20240503 14:15:18.623090 25947 tablet.cc:2369] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:15:18.949092 26377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:18.949496 26377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:18.957254 26377 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:18.972138 26367 consensus_queue.cc:1035] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 14:15:18.993425 26367 consensus_queue.cc:1035] T b70ada8e0ac24fd5a7a4d7760051d197 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:19.115926 26293 raft_consensus.cc:1261] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:19.116717 26367 consensus_queue.cc:1035] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:19.179584 26390 mvcc.cc:205] Tried to move back new op lower bound from 7023598465490337792 to 7023598463242862592. Current Snapshot: MvccSnapshot[applied={T|T < 7023598465490337792}]
INSERT report
rows total: 2000
time total: 194.3 ms
time per row: 0.0971502 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:15:20.162840 25757 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38548:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:20.163606 25757 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 14:15:20.172605 25879 tablet_service.cc:1467] Processing CreateTablet for tablet 5eba7a96b98a4c5bbef5d3b386021dac (DEFAULT_TABLE table=rebalance_test_table_1 [id=417923559eac4111af5209e6035bd69e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:20.172809 26275 tablet_service.cc:1467] Processing CreateTablet for tablet 5eba7a96b98a4c5bbef5d3b386021dac (DEFAULT_TABLE table=rebalance_test_table_1 [id=417923559eac4111af5209e6035bd69e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:20.173115 25879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eba7a96b98a4c5bbef5d3b386021dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:20.173220 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eba7a96b98a4c5bbef5d3b386021dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:20.173396 26273 tablet_service.cc:1467] Processing CreateTablet for tablet cd66d059a2124fa985e0b12f6c9fa37e (DEFAULT_TABLE table=rebalance_test_table_1 [id=417923559eac4111af5209e6035bd69e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:20.173676 26274 tablet_service.cc:1467] Processing CreateTablet for tablet 11c6a3915f864de49f2bb6b03a4654b7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=417923559eac4111af5209e6035bd69e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:20.174153 26274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c6a3915f864de49f2bb6b03a4654b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:20.174271 25880 tablet_service.cc:1467] Processing CreateTablet for tablet 11c6a3915f864de49f2bb6b03a4654b7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=417923559eac4111af5209e6035bd69e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:20.174542 25880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c6a3915f864de49f2bb6b03a4654b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:20.177763 25881 tablet_service.cc:1467] Processing CreateTablet for tablet cd66d059a2124fa985e0b12f6c9fa37e (DEFAULT_TABLE table=rebalance_test_table_1 [id=417923559eac4111af5209e6035bd69e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:20.178125 25881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd66d059a2124fa985e0b12f6c9fa37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:20.179961 26273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd66d059a2124fa985e0b12f6c9fa37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:20.186717 26429 tablet_bootstrap.cc:492] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c: Bootstrap starting.
I20240503 14:15:20.188553 26429 tablet_bootstrap.cc:654] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:20.190485 26428 tablet_bootstrap.cc:492] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe: Bootstrap starting.
I20240503 14:15:20.191759 26429 tablet_bootstrap.cc:492] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c: No bootstrap required, opened a new log
I20240503 14:15:20.191928 26429 ts_tablet_manager.cc:1367] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:15:20.192222 26428 tablet_bootstrap.cc:654] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:20.192703 26429 raft_consensus.cc:348] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:20.192929 26429 raft_consensus.cc:374] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:20.193024 26429 raft_consensus.cc:729] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Initialized, Role: FOLLOWER
I20240503 14:15:20.193400 26429 consensus_queue.cc:260] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:20.194165 26429 ts_tablet_manager.cc:1398] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:20.194448 26429 tablet_bootstrap.cc:492] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c: Bootstrap starting.
I20240503 14:15:20.194552 26428 tablet_bootstrap.cc:492] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe: No bootstrap required, opened a new log
I20240503 14:15:20.194695 26428 ts_tablet_manager.cc:1367] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:20.195425 26428 raft_consensus.cc:348] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:20.195619 26428 raft_consensus.cc:374] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:20.195760 26428 raft_consensus.cc:729] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Initialized, Role: FOLLOWER
I20240503 14:15:20.195900 26429 tablet_bootstrap.cc:654] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:20.196014 26428 consensus_queue.cc:260] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:20.196564 26428 ts_tablet_manager.cc:1398] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:20.196870 26428 tablet_bootstrap.cc:492] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe: Bootstrap starting.
I20240503 14:15:20.198391 26428 tablet_bootstrap.cc:654] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:20.198791 26429 tablet_bootstrap.cc:492] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c: No bootstrap required, opened a new log
I20240503 14:15:20.198951 26429 ts_tablet_manager.cc:1367] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:15:20.199694 26429 raft_consensus.cc:348] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.199992 26429 raft_consensus.cc:374] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:20.200120 26429 raft_consensus.cc:729] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Initialized, Role: FOLLOWER
I20240503 14:15:20.200454 26429 consensus_queue.cc:260] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.201207 26429 ts_tablet_manager.cc:1398] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:20.201498 26429 tablet_bootstrap.cc:492] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c: Bootstrap starting.
I20240503 14:15:20.202103 26428 tablet_bootstrap.cc:492] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe: No bootstrap required, opened a new log
I20240503 14:15:20.202260 26428 ts_tablet_manager.cc:1367] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:15:20.202948 26428 raft_consensus.cc:348] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.203145 26428 raft_consensus.cc:374] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:20.203233 26429 tablet_bootstrap.cc:654] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:20.203230 26428 raft_consensus.cc:729] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Initialized, Role: FOLLOWER
I20240503 14:15:20.203464 26428 consensus_queue.cc:260] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.204006 26428 ts_tablet_manager.cc:1398] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:20.204743 26428 tablet_bootstrap.cc:492] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe: Bootstrap starting.
I20240503 14:15:20.205726 26429 tablet_bootstrap.cc:492] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c: No bootstrap required, opened a new log
I20240503 14:15:20.205862 26429 ts_tablet_manager.cc:1367] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:20.206338 26428 tablet_bootstrap.cc:654] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:20.206444 26429 raft_consensus.cc:348] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.206580 26429 raft_consensus.cc:374] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:20.206634 26429 raft_consensus.cc:729] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Initialized, Role: FOLLOWER
I20240503 14:15:20.206823 26429 consensus_queue.cc:260] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.207257 26429 ts_tablet_manager.cc:1398] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:15:20.208166 26428 tablet_bootstrap.cc:492] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe: No bootstrap required, opened a new log
I20240503 14:15:20.208309 26428 ts_tablet_manager.cc:1367] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:20.208912 26428 raft_consensus.cc:348] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.209084 26428 raft_consensus.cc:374] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:20.209174 26428 raft_consensus.cc:729] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Initialized, Role: FOLLOWER
I20240503 14:15:20.209444 26428 consensus_queue.cc:260] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.210121 26428 ts_tablet_manager.cc:1398] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:20.390033 26373 raft_consensus.cc:482] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:20.390347 26373 raft_consensus.cc:504] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:20.391009 26373 leader_election.cc:288] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:20.391697 26293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c6a3915f864de49f2bb6b03a4654b7" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5deb936bb744b392e3a28be25aa55c" is_pre_election: true
I20240503 14:15:20.391942 26293 raft_consensus.cc:2454] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 0.
I20240503 14:15:20.392400 25835 leader_election.cc:302] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:20.392741 26373 raft_consensus.cc:2781] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:20.392870 26373 raft_consensus.cc:482] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:20.392978 26373 raft_consensus.cc:3037] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:20.395278 26373 raft_consensus.cc:504] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:20.395859 26373 leader_election.cc:288] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 election: Requested vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:20.396442 26293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c6a3915f864de49f2bb6b03a4654b7" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5deb936bb744b392e3a28be25aa55c"
I20240503 14:15:20.396644 26293 raft_consensus.cc:3037] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:20.398754 26293 raft_consensus.cc:2454] T 11c6a3915f864de49f2bb6b03a4654b7 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 1.
I20240503 14:15:20.399199 25835 leader_election.cc:302] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:20.399523 26373 raft_consensus.cc:2781] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:20.399870 26373 raft_consensus.cc:686] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Becoming Leader. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Running, Role: LEADER
I20240503 14:15:20.400157 26373 consensus_queue.cc:237] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:20.402544 25757 catalog_manager.cc:5484] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dd1204c91174803920e3a83d494a1fe (127.10.145.1). New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:20.469177 26373 raft_consensus.cc:482] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:20.469501 26373 raft_consensus.cc:504] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.470077 26373 leader_election.cc:288] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:20.470764 26293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd66d059a2124fa985e0b12f6c9fa37e" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5deb936bb744b392e3a28be25aa55c" is_pre_election: true
I20240503 14:15:20.470978 26293 raft_consensus.cc:2454] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 0.
I20240503 14:15:20.471391 25835 leader_election.cc:302] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:20.471696 26373 raft_consensus.cc:2781] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:20.471791 26373 raft_consensus.cc:482] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:20.471889 26373 raft_consensus.cc:3037] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:20.473910 26373 raft_consensus.cc:504] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.474391 26373 leader_election.cc:288] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 election: Requested vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:20.474943 26293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd66d059a2124fa985e0b12f6c9fa37e" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5deb936bb744b392e3a28be25aa55c"
I20240503 14:15:20.475155 26293 raft_consensus.cc:3037] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:20.477063 26293 raft_consensus.cc:2454] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 1.
I20240503 14:15:20.477527 25835 leader_election.cc:302] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:20.477830 26373 raft_consensus.cc:2781] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:20.477993 26373 raft_consensus.cc:686] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Becoming Leader. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Running, Role: LEADER
I20240503 14:15:20.478226 26373 consensus_queue.cc:237] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.480595 25757 catalog_manager.cc:5484] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dd1204c91174803920e3a83d494a1fe (127.10.145.1). New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:20.656128 26432 raft_consensus.cc:482] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:20.656390 26432 raft_consensus.cc:504] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.657931 26432 leader_election.cc:288] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:20.664217 25901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eba7a96b98a4c5bbef5d3b386021dac" candidate_uuid: "0d5deb936bb744b392e3a28be25aa55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd1204c91174803920e3a83d494a1fe" is_pre_election: true
I20240503 14:15:20.664592 25901 raft_consensus.cc:2454] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d5deb936bb744b392e3a28be25aa55c in term 0.
I20240503 14:15:20.665258 26228 leader_election.cc:302] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:20.665712 26432 raft_consensus.cc:2781] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:20.665843 26432 raft_consensus.cc:482] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:20.665938 26432 raft_consensus.cc:3037] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:20.667918 26432 raft_consensus.cc:504] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.668427 26432 leader_election.cc:288] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 election: Requested vote from peers 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:20.669037 25901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eba7a96b98a4c5bbef5d3b386021dac" candidate_uuid: "0d5deb936bb744b392e3a28be25aa55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd1204c91174803920e3a83d494a1fe"
I20240503 14:15:20.669217 25901 raft_consensus.cc:3037] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:20.671320 25901 raft_consensus.cc:2454] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5deb936bb744b392e3a28be25aa55c in term 1.
I20240503 14:15:20.671756 26228 leader_election.cc:302] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:20.672056 26432 raft_consensus.cc:2781] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:20.672431 26432 raft_consensus.cc:686] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Becoming Leader. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Running, Role: LEADER
I20240503 14:15:20.672955 26432 consensus_queue.cc:237] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:20.676333 25757 catalog_manager.cc:5484] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4). New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:20.955595 26373 consensus_queue.cc:1035] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:20.961959 26437 consensus_queue.cc:1035] T 11c6a3915f864de49f2bb6b03a4654b7 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:21.103907 26432 consensus_queue.cc:1035] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:21.111377 26444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:21.111716 26444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:21.119346 26444 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 210.342 ms
time per row: 0.105171 ms
I20240503 14:15:21.483935 25755 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38548:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:21.484777 25755 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 14:15:21.493613 25880 tablet_service.cc:1467] Processing CreateTablet for tablet 4dee80b89ad7448e87bbd976b78a24d0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=71eedb8dc1b14d2eb0d7ce00ccc4423e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:21.493789 26273 tablet_service.cc:1467] Processing CreateTablet for tablet 4dee80b89ad7448e87bbd976b78a24d0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=71eedb8dc1b14d2eb0d7ce00ccc4423e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:21.494148 25880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dee80b89ad7448e87bbd976b78a24d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:21.494177 26273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dee80b89ad7448e87bbd976b78a24d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:21.494676 25879 tablet_service.cc:1467] Processing CreateTablet for tablet d492084b25d54e28a9cbf7e4fe8b18ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=71eedb8dc1b14d2eb0d7ce00ccc4423e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:21.494951 25881 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc68ced13df43679e4fc4e8aef08f97 (DEFAULT_TABLE table=rebalance_test_table_2 [id=71eedb8dc1b14d2eb0d7ce00ccc4423e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:21.495102 25879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d492084b25d54e28a9cbf7e4fe8b18ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:21.495283 25881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc68ced13df43679e4fc4e8aef08f97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:21.496906 26274 tablet_service.cc:1467] Processing CreateTablet for tablet d492084b25d54e28a9cbf7e4fe8b18ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=71eedb8dc1b14d2eb0d7ce00ccc4423e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:21.496928 26275 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc68ced13df43679e4fc4e8aef08f97 (DEFAULT_TABLE table=rebalance_test_table_2 [id=71eedb8dc1b14d2eb0d7ce00ccc4423e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:21.497365 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc68ced13df43679e4fc4e8aef08f97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:21.501385 26274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d492084b25d54e28a9cbf7e4fe8b18ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:21.502121 26496 tablet_bootstrap.cc:492] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe: Bootstrap starting.
I20240503 14:15:21.504123 26496 tablet_bootstrap.cc:654] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:21.509003 26495 tablet_bootstrap.cc:492] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c: Bootstrap starting.
I20240503 14:15:21.511569 26495 tablet_bootstrap.cc:654] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:21.513423 26496 tablet_bootstrap.cc:492] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe: No bootstrap required, opened a new log
I20240503 14:15:21.513593 26496 ts_tablet_manager.cc:1367] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20240503 14:15:21.513909 26495 tablet_bootstrap.cc:492] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c: No bootstrap required, opened a new log
I20240503 14:15:21.514048 26495 ts_tablet_manager.cc:1367] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:15:21.514375 26496 raft_consensus.cc:348] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:21.514592 26496 raft_consensus.cc:374] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:21.514681 26496 raft_consensus.cc:729] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Initialized, Role: FOLLOWER
I20240503 14:15:21.514755 26495 raft_consensus.cc:348] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:21.514946 26495 raft_consensus.cc:374] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:21.514890 26496 consensus_queue.cc:260] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:21.515038 26495 raft_consensus.cc:729] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Initialized, Role: FOLLOWER
I20240503 14:15:21.515247 26495 consensus_queue.cc:260] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:21.515724 26495 ts_tablet_manager.cc:1398] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:21.516034 26495 tablet_bootstrap.cc:492] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: Bootstrap starting.
I20240503 14:15:21.516050 26496 ts_tablet_manager.cc:1398] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:21.516348 26496 tablet_bootstrap.cc:492] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe: Bootstrap starting.
I20240503 14:15:21.517669 26496 tablet_bootstrap.cc:654] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:21.519905 26496 tablet_bootstrap.cc:492] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe: No bootstrap required, opened a new log
I20240503 14:15:21.520107 26496 ts_tablet_manager.cc:1367] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:21.520313 26495 tablet_bootstrap.cc:654] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:21.520865 26496 raft_consensus.cc:348] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.521025 26496 raft_consensus.cc:374] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:21.521142 26496 raft_consensus.cc:729] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Initialized, Role: FOLLOWER
I20240503 14:15:21.521422 26496 consensus_queue.cc:260] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.521963 26496 ts_tablet_manager.cc:1398] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:21.522099 26495 tablet_bootstrap.cc:492] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: No bootstrap required, opened a new log
I20240503 14:15:21.522238 26495 ts_tablet_manager.cc:1367] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:15:21.522266 26496 tablet_bootstrap.cc:492] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe: Bootstrap starting.
I20240503 14:15:21.522871 26495 raft_consensus.cc:348] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.523032 26495 raft_consensus.cc:374] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:21.523129 26495 raft_consensus.cc:729] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Initialized, Role: FOLLOWER
I20240503 14:15:21.523629 26495 consensus_queue.cc:260] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.523844 26496 tablet_bootstrap.cc:654] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:21.524863 26495 ts_tablet_manager.cc:1398] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:15:21.525204 26495 tablet_bootstrap.cc:492] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c: Bootstrap starting.
I20240503 14:15:21.527285 26495 tablet_bootstrap.cc:654] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:21.530514 26496 tablet_bootstrap.cc:492] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe: No bootstrap required, opened a new log
I20240503 14:15:21.530670 26496 ts_tablet_manager.cc:1367] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20240503 14:15:21.531270 26496 raft_consensus.cc:348] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.531433 26496 raft_consensus.cc:374] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:21.531524 26496 raft_consensus.cc:729] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Initialized, Role: FOLLOWER
I20240503 14:15:21.531723 26496 consensus_queue.cc:260] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.532187 26496 ts_tablet_manager.cc:1398] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:15:21.533079 26495 tablet_bootstrap.cc:492] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c: No bootstrap required, opened a new log
I20240503 14:15:21.533270 26495 ts_tablet_manager.cc:1367] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:15:21.533995 26495 raft_consensus.cc:348] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.534185 26495 raft_consensus.cc:374] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:21.534282 26495 raft_consensus.cc:729] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Initialized, Role: FOLLOWER
I20240503 14:15:21.534516 26495 consensus_queue.cc:260] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.535071 26495 ts_tablet_manager.cc:1398] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:21.563187 26442 raft_consensus.cc:482] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:21.563385 26442 raft_consensus.cc:504] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.564065 26442 leader_election.cc:288] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:21.564821 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc68ced13df43679e4fc4e8aef08f97" candidate_uuid: "0d5deb936bb744b392e3a28be25aa55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd1204c91174803920e3a83d494a1fe" is_pre_election: true
I20240503 14:15:21.565070 25900 raft_consensus.cc:2454] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d5deb936bb744b392e3a28be25aa55c in term 0.
I20240503 14:15:21.565558 26228 leader_election.cc:302] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:21.565862 26442 raft_consensus.cc:2781] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:21.565992 26442 raft_consensus.cc:482] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:21.566084 26442 raft_consensus.cc:3037] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:21.568229 26442 raft_consensus.cc:504] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.568774 26442 leader_election.cc:288] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 election: Requested vote from peers 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:21.569512 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc68ced13df43679e4fc4e8aef08f97" candidate_uuid: "0d5deb936bb744b392e3a28be25aa55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd1204c91174803920e3a83d494a1fe"
I20240503 14:15:21.569707 25900 raft_consensus.cc:3037] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:21.571961 25900 raft_consensus.cc:2454] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5deb936bb744b392e3a28be25aa55c in term 1.
I20240503 14:15:21.572357 26228 leader_election.cc:302] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:21.572691 26442 raft_consensus.cc:2781] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:21.572840 26442 raft_consensus.cc:686] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Becoming Leader. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Running, Role: LEADER
I20240503 14:15:21.573117 26442 consensus_queue.cc:237] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.575773 25752 catalog_manager.cc:5484] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4). New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:21.612468 26442 raft_consensus.cc:482] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:21.612701 26442 raft_consensus.cc:504] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.613400 26442 leader_election.cc:288] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:21.614099 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d492084b25d54e28a9cbf7e4fe8b18ec" candidate_uuid: "0d5deb936bb744b392e3a28be25aa55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd1204c91174803920e3a83d494a1fe" is_pre_election: true
I20240503 14:15:21.614336 25900 raft_consensus.cc:2454] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d5deb936bb744b392e3a28be25aa55c in term 0.
I20240503 14:15:21.614758 26228 leader_election.cc:302] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:21.615067 26442 raft_consensus.cc:2781] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:21.615172 26442 raft_consensus.cc:482] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:21.615278 26442 raft_consensus.cc:3037] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:21.617172 26442 raft_consensus.cc:504] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.617697 26442 leader_election.cc:288] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 election: Requested vote from peers 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:21.618237 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d492084b25d54e28a9cbf7e4fe8b18ec" candidate_uuid: "0d5deb936bb744b392e3a28be25aa55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd1204c91174803920e3a83d494a1fe"
I20240503 14:15:21.618454 25900 raft_consensus.cc:3037] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:21.620337 25900 raft_consensus.cc:2454] T d492084b25d54e28a9cbf7e4fe8b18ec P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5deb936bb744b392e3a28be25aa55c in term 1.
I20240503 14:15:21.620777 26228 leader_election.cc:302] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:21.621089 26442 raft_consensus.cc:2781] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:21.621299 26442 raft_consensus.cc:686] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Becoming Leader. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Running, Role: LEADER
I20240503 14:15:21.621670 26442 consensus_queue.cc:237] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } }
I20240503 14:15:21.624300 25752 catalog_manager.cc:5484] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4). New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:21.763341 26442 raft_consensus.cc:482] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:21.763568 26442 raft_consensus.cc:504] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:21.764153 26442 leader_election.cc:288] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:21.764799 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dee80b89ad7448e87bbd976b78a24d0" candidate_uuid: "0d5deb936bb744b392e3a28be25aa55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd1204c91174803920e3a83d494a1fe" is_pre_election: true
I20240503 14:15:21.765015 25900 raft_consensus.cc:2454] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d5deb936bb744b392e3a28be25aa55c in term 0.
I20240503 14:15:21.765517 26228 leader_election.cc:302] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:21.765826 26442 raft_consensus.cc:2781] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:21.765955 26442 raft_consensus.cc:482] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:21.766038 26442 raft_consensus.cc:3037] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:21.767992 26442 raft_consensus.cc:504] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:21.768468 26442 leader_election.cc:288] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 election: Requested vote from peers 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:21.769080 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dee80b89ad7448e87bbd976b78a24d0" candidate_uuid: "0d5deb936bb744b392e3a28be25aa55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd1204c91174803920e3a83d494a1fe"
I20240503 14:15:21.769266 25900 raft_consensus.cc:3037] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:21.771277 25900 raft_consensus.cc:2454] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5deb936bb744b392e3a28be25aa55c in term 1.
I20240503 14:15:21.771709 26228 leader_election.cc:302] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:21.771986 26442 raft_consensus.cc:2781] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:21.772145 26442 raft_consensus.cc:686] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Becoming Leader. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Running, Role: LEADER
I20240503 14:15:21.772476 26442 consensus_queue.cc:237] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } }
I20240503 14:15:21.775046 25752 catalog_manager.cc:5484] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4). New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:22.008934 26442 consensus_queue.cc:1035] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:22.154563 26432 consensus_queue.cc:1035] T d492084b25d54e28a9cbf7e4fe8b18ec P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:22.158000 26506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:22.158564 26506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:22.166999 26506 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:15:22.298765 26432 consensus_queue.cc:1035] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 195.05 ms
time per row: 0.0975249 ms
I20240503 14:15:22.506863 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.2:35219
--local_ip_for_outbound_sockets=127.10.145.2
--tserver_master_addrs=127.10.145.62:33795
--webserver_port=44683
--webserver_interface=127.10.145.2
--builtin_ntp_servers=127.10.145.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:15:22.860220 26550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:22.860558 26550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:22.860679 26550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:22.868431 26550 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:22.868580 26550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:22.868728 26550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.2
I20240503 14:15:22.878422 26550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.2:35219
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.145.2
--webserver_port=44683
--tserver_master_addrs=127.10.145.62:33795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.2
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:22.879850 26550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:22.881729 26550 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:22.894776 26556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:22.895079 26555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:22.895335 26558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:22.895742 26550 server_base.cc:1034] running on GCE node
I20240503 14:15:22.896430 26550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:22.897493 26550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:22.898694 26550 hybrid_clock.cc:648] HybridClock initialized: now 1714745722898638 us; error 58 us; skew 500 ppm
I20240503 14:15:22.899022 26550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:22.901615 26550 webserver.cc:460] Webserver started at http://127.10.145.2:44683/ using document root <none> and password file <none>
I20240503 14:15:22.902447 26550 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:22.902590 26550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:22.908627 26550 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 14:15:22.911650 26564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:22.913214 26550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:22.913527 26550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a680f58704cf4c979e241a706d51b972"
format_stamp: "Formatted at 2024-05-03 14:15:13 on dist-test-slave-v549"
I20240503 14:15:22.914391 26550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:22.931470 26550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:22.932385 26550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:22.932721 26550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:22.933889 26550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:22.935554 26550 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:22.935628 26550 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:22.935748 26550 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:22.935796 26550 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:22.966358 26550 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.2:35219
I20240503 14:15:22.966398 26676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.2:35219 every 8 connection(s)
I20240503 14:15:22.967986 26550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:15:22.977764 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 26550
I20240503 14:15:22.978590 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.3:46613
--local_ip_for_outbound_sockets=127.10.145.3
--tserver_master_addrs=127.10.145.62:33795
--webserver_port=41577
--webserver_interface=127.10.145.3
--builtin_ntp_servers=127.10.145.20:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:22.982640 26677 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33795
I20240503 14:15:22.983045 26677 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:22.984160 26677 heartbeater.cc:511] Master 127.10.145.62:33795 requested a full tablet report, sending...
I20240503 14:15:22.986261 25759 ts_manager.cc:194] Re-registered known tserver with Master: a680f58704cf4c979e241a706d51b972 (127.10.145.2:35219)
I20240503 14:15:22.987166 25759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.2:52339
W20240503 14:15:23.345861 26681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:23.346215 26681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:23.346343 26681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:23.354108 26681 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:23.354280 26681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:23.354444 26681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.3
I20240503 14:15:23.363891 26681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.3:46613
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.145.3
--webserver_port=41577
--tserver_master_addrs=127.10.145.62:33795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.3
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:23.365445 26681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:23.367453 26681 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:23.380270 26686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:23.380463 26687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:23.380811 26689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:23.381217 26681 server_base.cc:1034] running on GCE node
I20240503 14:15:23.381917 26681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:23.382999 26681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:23.384235 26681 hybrid_clock.cc:648] HybridClock initialized: now 1714745723384149 us; error 104 us; skew 500 ppm
I20240503 14:15:23.384712 26681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:23.387387 26681 webserver.cc:460] Webserver started at http://127.10.145.3:41577/ using document root <none> and password file <none>
I20240503 14:15:23.388164 26681 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:23.388286 26681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:23.394317 26681 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 14:15:23.397620 26695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:23.399127 26681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:15:23.399314 26681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "171289531539427c84649caa8aa7f86e"
format_stamp: "Formatted at 2024-05-03 14:15:13 on dist-test-slave-v549"
I20240503 14:15:23.400188 26681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:23.424814 26681 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:23.425891 26681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:23.426193 26681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:23.427254 26681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:23.428893 26681 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:23.428970 26681 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:23.429091 26681 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:23.429137 26681 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:23.461083 26681 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.3:46613
I20240503 14:15:23.461163 26807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.3:46613 every 8 connection(s)
I20240503 14:15:23.463213 26681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:15:23.473101 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 26681
I20240503 14:15:23.477890 26808 heartbeater.cc:347] Connected to a master server at 127.10.145.62:33795
I20240503 14:15:23.478282 26808 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:23.479367 26808 heartbeater.cc:511] Master 127.10.145.62:33795 requested a full tablet report, sending...
I20240503 14:15:23.481452 25759 ts_manager.cc:194] Re-registered known tserver with Master: 171289531539427c84649caa8aa7f86e (127.10.145.3:46613)
I20240503 14:15:23.482396 25759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.3:38677
I20240503 14:15:23.990849 26677 heartbeater.cc:503] Master 127.10.145.62:33795 was elected leader, sending a full tablet report...
I20240503 14:15:24.021559 26743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:24.033250 25879 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:24.038758 26612 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:24.076727 26274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:24.237553 26743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:24.241740 26612 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:24.243203 26274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:24.243441 25879 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:24.436312 26274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:24.436987 26743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:24.445500 25879 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:24.453738 26612 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:24.485929 26808 heartbeater.cc:503] Master 127.10.145.62:33795 was elected leader, sending a full tablet report...
I20240503 14:15:24.605686 26274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:24.607584 26743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:24.610687 25879 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:24.621572 26612 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:24.684990 26293 consensus_queue.cc:237] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } }
I20240503 14:15:24.687641 25901 raft_consensus.cc:1261] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:24.688251 26442 consensus_queue.cc:1035] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:24.691512 26949 raft_consensus.cc:2932] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 171289531539427c84649caa8aa7f86e (127.10.145.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } } }
I20240503 14:15:24.692283 25901 raft_consensus.cc:2932] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 171289531539427c84649caa8aa7f86e (127.10.145.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } } }
W20240503 14:15:24.695420 26230 consensus_peers.cc:485] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c -> Peer 171289531539427c84649caa8aa7f86e (127.10.145.3:46613): Couldn't send request to peer 171289531539427c84649caa8aa7f86e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bc68ced13df43679e4fc4e8aef08f97. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:24.697440 25757 catalog_manager.cc:5484] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe reported cstate change: config changed from index -1 to 4, NON_VOTER 171289531539427c84649caa8aa7f86e (127.10.145.3) added. New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } } }
I20240503 14:15:24.709615 26293 consensus_queue.cc:237] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } }
I20240503 14:15:24.711983 25901 raft_consensus.cc:1261] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:24.712723 26442 consensus_queue.cc:1035] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:24.715828 26949 raft_consensus.cc:2932] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a680f58704cf4c979e241a706d51b972 (127.10.145.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } } }
I20240503 14:15:24.716645 25901 raft_consensus.cc:2932] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a680f58704cf4c979e241a706d51b972 (127.10.145.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } } }
W20240503 14:15:24.720903 26229 consensus_peers.cc:485] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c -> Peer a680f58704cf4c979e241a706d51b972 (127.10.145.2:35219): Couldn't send request to peer a680f58704cf4c979e241a706d51b972. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5eba7a96b98a4c5bbef5d3b386021dac. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:24.722793 25752 catalog_manager.cc:5484] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe reported cstate change: config changed from index -1 to 4, NON_VOTER a680f58704cf4c979e241a706d51b972 (127.10.145.2) added. New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } } }
I20240503 14:15:24.733979 25901 consensus_queue.cc:237] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } }
I20240503 14:15:24.736876 26293 raft_consensus.cc:1261] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:24.737617 26812 consensus_queue.cc:1035] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:24.740988 26969 raft_consensus.cc:2932] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 171289531539427c84649caa8aa7f86e (127.10.145.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } } }
I20240503 14:15:24.741653 26293 raft_consensus.cc:2932] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 171289531539427c84649caa8aa7f86e (127.10.145.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } } }
W20240503 14:15:24.744485 25836 consensus_peers.cc:485] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe -> Peer 171289531539427c84649caa8aa7f86e (127.10.145.3:46613): Couldn't send request to peer 171289531539427c84649caa8aa7f86e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd66d059a2124fa985e0b12f6c9fa37e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:24.746670 25757 catalog_manager.cc:5484] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe reported cstate change: config changed from index -1 to 4, NON_VOTER 171289531539427c84649caa8aa7f86e (127.10.145.3) added. New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:24.760144 26293 consensus_queue.cc:237] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } }
W20240503 14:15:24.762799 26229 consensus_peers.cc:485] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c -> Peer a680f58704cf4c979e241a706d51b972 (127.10.145.2:35219): Couldn't send request to peer a680f58704cf4c979e241a706d51b972. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dee80b89ad7448e87bbd976b78a24d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:24.762856 25901 raft_consensus.cc:1261] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:24.763415 26949 consensus_queue.cc:1035] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:24.766280 26442 raft_consensus.cc:2932] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a680f58704cf4c979e241a706d51b972 (127.10.145.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } } }
I20240503 14:15:24.766963 25901 raft_consensus.cc:2932] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a680f58704cf4c979e241a706d51b972 (127.10.145.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } } }
I20240503 14:15:24.771883 25759 catalog_manager.cc:5484] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c reported cstate change: config changed from index -1 to 4, NON_VOTER a680f58704cf4c979e241a706d51b972 (127.10.145.2) added. New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:24.785041 25901 consensus_queue.cc:237] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } }
I20240503 14:15:24.787297 26293 raft_consensus.cc:1261] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:24.787881 26812 consensus_queue.cc:1035] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:24.790766 26969 raft_consensus.cc:2932] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a680f58704cf4c979e241a706d51b972 (127.10.145.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } } }
W20240503 14:15:24.792788 25835 consensus_peers.cc:485] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe -> Peer a680f58704cf4c979e241a706d51b972 (127.10.145.2:35219): Couldn't send request to peer a680f58704cf4c979e241a706d51b972. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44de804572554d56becf3387e1031e1b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:24.792596 26293 raft_consensus.cc:2932] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a680f58704cf4c979e241a706d51b972 (127.10.145.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } } }
I20240503 14:15:24.795246 25757 catalog_manager.cc:5484] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe reported cstate change: config changed from index -1 to 4, NON_VOTER a680f58704cf4c979e241a706d51b972 (127.10.145.2) added. New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:24.817238 25901 consensus_queue.cc:237] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } }
I20240503 14:15:24.819693 26293 raft_consensus.cc:1261] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:15:24.819849 25836 consensus_peers.cc:485] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe -> Peer 171289531539427c84649caa8aa7f86e (127.10.145.3:46613): Couldn't send request to peer 171289531539427c84649caa8aa7f86e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 209eb1a4b190422ea47b12f96755f064. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:24.820268 26969 consensus_queue.cc:1035] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:24.822999 26812 raft_consensus.cc:2932] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 171289531539427c84649caa8aa7f86e (127.10.145.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } } }
I20240503 14:15:24.823689 26293 raft_consensus.cc:2932] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 171289531539427c84649caa8aa7f86e (127.10.145.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } } }
I20240503 14:15:24.829077 25759 catalog_manager.cc:5484] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe reported cstate change: config changed from index -1 to 4, NON_VOTER 171289531539427c84649caa8aa7f86e (127.10.145.3) added. New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:25.097375 27028 ts_tablet_manager.cc:902] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: Initiating tablet copy from peer 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:25.099445 27028 tablet_copy_client.cc:308] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:37471
I20240503 14:15:25.107556 26315 tablet_copy_service.cc:140] P 0d5deb936bb744b392e3a28be25aa55c: Received BeginTabletCopySession request for tablet 1bc68ced13df43679e4fc4e8aef08f97 from peer 171289531539427c84649caa8aa7f86e ({username='slave'} at 127.10.145.3:49821)
I20240503 14:15:25.107896 26315 tablet_copy_service.cc:161] P 0d5deb936bb744b392e3a28be25aa55c: Beginning new tablet copy session on tablet 1bc68ced13df43679e4fc4e8aef08f97 from peer 171289531539427c84649caa8aa7f86e at {username='slave'} at 127.10.145.3:49821: session id = 171289531539427c84649caa8aa7f86e-1bc68ced13df43679e4fc4e8aef08f97
I20240503 14:15:25.111477 26315 tablet_copy_source_session.cc:205] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:25.117583 27028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc68ced13df43679e4fc4e8aef08f97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:25.128970 27028 tablet_copy_client.cc:783] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:25.129319 27028 tablet_copy_client.cc:647] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:25.131999 27028 tablet_copy_client.cc:523] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:25.136476 27028 tablet_bootstrap.cc:492] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: Bootstrap starting.
I20240503 14:15:25.149560 27028 log.cc:824] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:25.161883 27051 ts_tablet_manager.cc:902] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: Initiating tablet copy from peer 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:25.167356 27051 tablet_copy_client.cc:308] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:37471
I20240503 14:15:25.175024 26315 tablet_copy_service.cc:140] P 0d5deb936bb744b392e3a28be25aa55c: Received BeginTabletCopySession request for tablet 5eba7a96b98a4c5bbef5d3b386021dac from peer a680f58704cf4c979e241a706d51b972 ({username='slave'} at 127.10.145.2:35199)
I20240503 14:15:25.175258 26315 tablet_copy_service.cc:161] P 0d5deb936bb744b392e3a28be25aa55c: Beginning new tablet copy session on tablet 5eba7a96b98a4c5bbef5d3b386021dac from peer a680f58704cf4c979e241a706d51b972 at {username='slave'} at 127.10.145.2:35199: session id = a680f58704cf4c979e241a706d51b972-5eba7a96b98a4c5bbef5d3b386021dac
I20240503 14:15:25.177266 26315 tablet_copy_source_session.cc:205] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:25.182955 27051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eba7a96b98a4c5bbef5d3b386021dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:25.189342 27066 ts_tablet_manager.cc:902] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972: Initiating tablet copy from peer 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471)
I20240503 14:15:25.198803 27066 tablet_copy_client.cc:308] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:37471
I20240503 14:15:25.199733 26315 tablet_copy_service.cc:140] P 0d5deb936bb744b392e3a28be25aa55c: Received BeginTabletCopySession request for tablet 4dee80b89ad7448e87bbd976b78a24d0 from peer a680f58704cf4c979e241a706d51b972 ({username='slave'} at 127.10.145.2:35199)
I20240503 14:15:25.199919 26315 tablet_copy_service.cc:161] P 0d5deb936bb744b392e3a28be25aa55c: Beginning new tablet copy session on tablet 4dee80b89ad7448e87bbd976b78a24d0 from peer a680f58704cf4c979e241a706d51b972 at {username='slave'} at 127.10.145.2:35199: session id = a680f58704cf4c979e241a706d51b972-4dee80b89ad7448e87bbd976b78a24d0
I20240503 14:15:25.200331 27051 tablet_copy_client.cc:783] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:25.200695 27051 tablet_copy_client.cc:647] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:25.202827 26315 tablet_copy_source_session.cc:205] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:25.203157 27051 tablet_copy_client.cc:523] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:25.204320 27066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dee80b89ad7448e87bbd976b78a24d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:25.204756 27028 tablet_bootstrap.cc:492] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:25.205385 27028 tablet_bootstrap.cc:492] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: Bootstrap complete.
I20240503 14:15:25.205926 27028 ts_tablet_manager.cc:1367] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: Time spent bootstrapping tablet: real 0.070s user 0.059s sys 0.012s
I20240503 14:15:25.207342 27051 tablet_bootstrap.cc:492] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: Bootstrap starting.
I20240503 14:15:25.212157 27066 tablet_copy_client.cc:783] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:25.212455 27066 tablet_copy_client.cc:647] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:25.212615 27071 ts_tablet_manager.cc:902] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e: Initiating tablet copy from peer 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:25.213052 27028 raft_consensus.cc:348] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } }
I20240503 14:15:25.213631 27028 raft_consensus.cc:729] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 171289531539427c84649caa8aa7f86e, State: Initialized, Role: LEARNER
I20240503 14:15:25.214314 27028 consensus_queue.cc:260] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } }
I20240503 14:15:25.214551 27066 tablet_copy_client.cc:523] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:25.218827 27066 tablet_bootstrap.cc:492] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972: Bootstrap starting.
I20240503 14:15:25.222481 27051 log.cc:824] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:25.223001 27028 ts_tablet_manager.cc:1398] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20240503 14:15:25.229758 26315 tablet_copy_service.cc:342] P 0d5deb936bb744b392e3a28be25aa55c: Request end of tablet copy session 171289531539427c84649caa8aa7f86e-1bc68ced13df43679e4fc4e8aef08f97 received from {username='slave'} at 127.10.145.3:49821
I20240503 14:15:25.229970 26315 tablet_copy_service.cc:433] P 0d5deb936bb744b392e3a28be25aa55c: ending tablet copy session 171289531539427c84649caa8aa7f86e-1bc68ced13df43679e4fc4e8aef08f97 on tablet 1bc68ced13df43679e4fc4e8aef08f97 with peer 171289531539427c84649caa8aa7f86e
I20240503 14:15:25.236817 27028 ts_tablet_manager.cc:902] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e: Initiating tablet copy from peer 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:25.237638 27071 tablet_copy_client.cc:308] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:45305
I20240503 14:15:25.246163 27028 tablet_copy_client.cc:308] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:45305
I20240503 14:15:25.249755 25921 tablet_copy_service.cc:140] P 6dd1204c91174803920e3a83d494a1fe: Received BeginTabletCopySession request for tablet cd66d059a2124fa985e0b12f6c9fa37e from peer 171289531539427c84649caa8aa7f86e ({username='slave'} at 127.10.145.3:44549)
I20240503 14:15:25.250090 25921 tablet_copy_service.cc:161] P 6dd1204c91174803920e3a83d494a1fe: Beginning new tablet copy session on tablet cd66d059a2124fa985e0b12f6c9fa37e from peer 171289531539427c84649caa8aa7f86e at {username='slave'} at 127.10.145.3:44549: session id = 171289531539427c84649caa8aa7f86e-cd66d059a2124fa985e0b12f6c9fa37e
I20240503 14:15:25.254148 25921 tablet_copy_source_session.cc:205] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:25.256779 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd66d059a2124fa985e0b12f6c9fa37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:25.257810 25920 tablet_copy_service.cc:140] P 6dd1204c91174803920e3a83d494a1fe: Received BeginTabletCopySession request for tablet 209eb1a4b190422ea47b12f96755f064 from peer 171289531539427c84649caa8aa7f86e ({username='slave'} at 127.10.145.3:44549)
I20240503 14:15:25.258028 25920 tablet_copy_service.cc:161] P 6dd1204c91174803920e3a83d494a1fe: Beginning new tablet copy session on tablet 209eb1a4b190422ea47b12f96755f064 from peer 171289531539427c84649caa8aa7f86e at {username='slave'} at 127.10.145.3:44549: session id = 171289531539427c84649caa8aa7f86e-209eb1a4b190422ea47b12f96755f064
I20240503 14:15:25.261309 25920 tablet_copy_source_session.cc:205] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:25.264828 27071 tablet_copy_client.cc:783] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:25.265254 27071 tablet_copy_client.cc:647] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:25.266120 27028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209eb1a4b190422ea47b12f96755f064. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:25.267823 27071 tablet_copy_client.cc:523] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:25.278846 27028 tablet_copy_client.cc:783] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:25.280490 27071 tablet_bootstrap.cc:492] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e: Bootstrap starting.
I20240503 14:15:25.280889 27028 tablet_copy_client.cc:647] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:25.282527 27066 tablet_bootstrap.cc:492] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:25.283203 27066 tablet_bootstrap.cc:492] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972: Bootstrap complete.
I20240503 14:15:25.283213 27028 tablet_copy_client.cc:523] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:25.283665 27051 tablet_bootstrap.cc:492] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:25.283761 27066 ts_tablet_manager.cc:1367] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972: Time spent bootstrapping tablet: real 0.065s user 0.051s sys 0.006s
I20240503 14:15:25.284104 27051 tablet_bootstrap.cc:492] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: Bootstrap complete.
I20240503 14:15:25.284358 27051 ts_tablet_manager.cc:1367] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: Time spent bootstrapping tablet: real 0.077s user 0.058s sys 0.008s
I20240503 14:15:25.287516 27028 tablet_bootstrap.cc:492] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e: Bootstrap starting.
I20240503 14:15:25.290681 27066 raft_consensus.cc:348] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } }
I20240503 14:15:25.290683 27051 raft_consensus.cc:348] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } }
I20240503 14:15:25.291329 27066 raft_consensus.cc:729] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a680f58704cf4c979e241a706d51b972, State: Initialized, Role: LEARNER
I20240503 14:15:25.291473 27051 raft_consensus.cc:729] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a680f58704cf4c979e241a706d51b972, State: Initialized, Role: LEARNER
I20240503 14:15:25.291982 27051 consensus_queue.cc:260] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } }
I20240503 14:15:25.291973 27066 consensus_queue.cc:260] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } }
I20240503 14:15:25.300210 27051 ts_tablet_manager.cc:1398] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20240503 14:15:25.300211 27066 ts_tablet_manager.cc:1398] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20240503 14:15:25.302556 26315 tablet_copy_service.cc:342] P 0d5deb936bb744b392e3a28be25aa55c: Request end of tablet copy session a680f58704cf4c979e241a706d51b972-4dee80b89ad7448e87bbd976b78a24d0 received from {username='slave'} at 127.10.145.2:35199
I20240503 14:15:25.302709 26315 tablet_copy_service.cc:433] P 0d5deb936bb744b392e3a28be25aa55c: ending tablet copy session a680f58704cf4c979e241a706d51b972-4dee80b89ad7448e87bbd976b78a24d0 on tablet 4dee80b89ad7448e87bbd976b78a24d0 with peer a680f58704cf4c979e241a706d51b972
I20240503 14:15:25.303054 26314 tablet_copy_service.cc:342] P 0d5deb936bb744b392e3a28be25aa55c: Request end of tablet copy session a680f58704cf4c979e241a706d51b972-5eba7a96b98a4c5bbef5d3b386021dac received from {username='slave'} at 127.10.145.2:35199
I20240503 14:15:25.303196 26314 tablet_copy_service.cc:433] P 0d5deb936bb744b392e3a28be25aa55c: ending tablet copy session a680f58704cf4c979e241a706d51b972-5eba7a96b98a4c5bbef5d3b386021dac on tablet 5eba7a96b98a4c5bbef5d3b386021dac with peer a680f58704cf4c979e241a706d51b972
I20240503 14:15:25.326797 27071 tablet_bootstrap.cc:492] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:25.327234 27071 tablet_bootstrap.cc:492] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e: Bootstrap complete.
I20240503 14:15:25.327541 27071 ts_tablet_manager.cc:1367] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e: Time spent bootstrapping tablet: real 0.047s user 0.038s sys 0.004s
I20240503 14:15:25.328457 27071 raft_consensus.cc:348] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } }
I20240503 14:15:25.328709 27071 raft_consensus.cc:729] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 171289531539427c84649caa8aa7f86e, State: Initialized, Role: LEARNER
I20240503 14:15:25.328907 27071 consensus_queue.cc:260] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } }
I20240503 14:15:25.329649 27071 ts_tablet_manager.cc:1398] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:15:25.330590 25921 tablet_copy_service.cc:342] P 6dd1204c91174803920e3a83d494a1fe: Request end of tablet copy session 171289531539427c84649caa8aa7f86e-cd66d059a2124fa985e0b12f6c9fa37e received from {username='slave'} at 127.10.145.3:44549
I20240503 14:15:25.330788 25921 tablet_copy_service.cc:433] P 6dd1204c91174803920e3a83d494a1fe: ending tablet copy session 171289531539427c84649caa8aa7f86e-cd66d059a2124fa985e0b12f6c9fa37e on tablet cd66d059a2124fa985e0b12f6c9fa37e with peer 171289531539427c84649caa8aa7f86e
I20240503 14:15:25.350983 27028 tablet_bootstrap.cc:492] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:25.351397 27028 tablet_bootstrap.cc:492] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e: Bootstrap complete.
I20240503 14:15:25.351657 27028 ts_tablet_manager.cc:1367] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e: Time spent bootstrapping tablet: real 0.064s user 0.037s sys 0.009s
I20240503 14:15:25.352351 27028 raft_consensus.cc:348] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } }
I20240503 14:15:25.352559 27028 raft_consensus.cc:729] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 171289531539427c84649caa8aa7f86e, State: Initialized, Role: LEARNER
I20240503 14:15:25.352751 27028 consensus_queue.cc:260] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: NON_VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: true } }
I20240503 14:15:25.353245 27028 ts_tablet_manager.cc:1398] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:15:25.354110 25921 tablet_copy_service.cc:342] P 6dd1204c91174803920e3a83d494a1fe: Request end of tablet copy session 171289531539427c84649caa8aa7f86e-209eb1a4b190422ea47b12f96755f064 received from {username='slave'} at 127.10.145.3:44549
I20240503 14:15:25.354244 25921 tablet_copy_service.cc:433] P 6dd1204c91174803920e3a83d494a1fe: ending tablet copy session 171289531539427c84649caa8aa7f86e-209eb1a4b190422ea47b12f96755f064 on tablet 209eb1a4b190422ea47b12f96755f064 with peer 171289531539427c84649caa8aa7f86e
I20240503 14:15:25.384601 27051 ts_tablet_manager.cc:902] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972: Initiating tablet copy from peer 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305)
I20240503 14:15:25.385962 27051 tablet_copy_client.cc:308] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:45305
I20240503 14:15:25.393092 25921 tablet_copy_service.cc:140] P 6dd1204c91174803920e3a83d494a1fe: Received BeginTabletCopySession request for tablet 44de804572554d56becf3387e1031e1b from peer a680f58704cf4c979e241a706d51b972 ({username='slave'} at 127.10.145.2:50435)
I20240503 14:15:25.393410 25921 tablet_copy_service.cc:161] P 6dd1204c91174803920e3a83d494a1fe: Beginning new tablet copy session on tablet 44de804572554d56becf3387e1031e1b from peer a680f58704cf4c979e241a706d51b972 at {username='slave'} at 127.10.145.2:50435: session id = a680f58704cf4c979e241a706d51b972-44de804572554d56becf3387e1031e1b
I20240503 14:15:25.395829 25921 tablet_copy_source_session.cc:205] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:25.397329 27051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44de804572554d56becf3387e1031e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:25.402418 27051 tablet_copy_client.cc:783] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:25.402714 27051 tablet_copy_client.cc:647] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:25.404603 27051 tablet_copy_client.cc:523] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:25.407691 27051 tablet_bootstrap.cc:492] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972: Bootstrap starting.
I20240503 14:15:25.444047 27051 tablet_bootstrap.cc:492] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:25.444517 27051 tablet_bootstrap.cc:492] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972: Bootstrap complete.
I20240503 14:15:25.444830 27051 ts_tablet_manager.cc:1367] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972: Time spent bootstrapping tablet: real 0.037s user 0.036s sys 0.000s
I20240503 14:15:25.445724 27051 raft_consensus.cc:348] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } }
I20240503 14:15:25.445986 27051 raft_consensus.cc:729] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a680f58704cf4c979e241a706d51b972, State: Initialized, Role: LEARNER
I20240503 14:15:25.446195 27051 consensus_queue.cc:260] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: NON_VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: true } }
I20240503 14:15:25.446674 27051 ts_tablet_manager.cc:1398] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:15:25.447491 25921 tablet_copy_service.cc:342] P 6dd1204c91174803920e3a83d494a1fe: Request end of tablet copy session a680f58704cf4c979e241a706d51b972-44de804572554d56becf3387e1031e1b received from {username='slave'} at 127.10.145.2:50435
I20240503 14:15:25.447631 25921 tablet_copy_service.cc:433] P 6dd1204c91174803920e3a83d494a1fe: ending tablet copy session a680f58704cf4c979e241a706d51b972-44de804572554d56becf3387e1031e1b on tablet 44de804572554d56becf3387e1031e1b with peer a680f58704cf4c979e241a706d51b972
I20240503 14:15:25.638746 26763 raft_consensus.cc:1203] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:25.640403 26763 raft_consensus.cc:1203] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:25.755386 26763 raft_consensus.cc:1203] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:25.762413 26632 raft_consensus.cc:1203] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:25.803344 26632 raft_consensus.cc:1203] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:25.996594 26632 raft_consensus.cc:1203] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:26.189353 27123 raft_consensus.cc:1050] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: attempting to promote NON_VOTER 171289531539427c84649caa8aa7f86e to VOTER
I20240503 14:15:26.190481 27123 consensus_queue.cc:237] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:26.193725 26763 raft_consensus.cc:1261] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.193982 25901 raft_consensus.cc:1261] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.194594 27123 consensus_queue.cc:1035] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.194972 27124 consensus_queue.cc:1035] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.197626 27124 raft_consensus.cc:2932] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.198269 25901 raft_consensus.cc:2932] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.203615 25759 catalog_manager.cc:5484] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c reported cstate change: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:26.205183 26763 raft_consensus.cc:2932] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.209858 27156 raft_consensus.cc:1050] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: attempting to promote NON_VOTER 171289531539427c84649caa8aa7f86e to VOTER
I20240503 14:15:26.210754 27156 consensus_queue.cc:237] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:26.214109 26763 raft_consensus.cc:1261] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.214655 26812 consensus_queue.cc:1035] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.214875 26293 raft_consensus.cc:1261] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.215782 26812 consensus_queue.cc:1035] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.217780 27125 raft_consensus.cc:2932] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.218557 26763 raft_consensus.cc:2932] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.219038 26293 raft_consensus.cc:2932] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.223120 25752 catalog_manager.cc:5484] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e reported cstate change: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.232947 27124 raft_consensus.cc:1050] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c: attempting to promote NON_VOTER a680f58704cf4c979e241a706d51b972 to VOTER
I20240503 14:15:26.233712 27124 consensus_queue.cc:237] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } }
I20240503 14:15:26.235927 25901 raft_consensus.cc:1261] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.236289 26632 raft_consensus.cc:1261] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.236603 27123 consensus_queue.cc:1035] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.237483 27123 consensus_queue.cc:1035] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.240020 27124 raft_consensus.cc:2932] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.241461 25901 raft_consensus.cc:2932] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.250867 25757 catalog_manager.cc:5484] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c reported cstate change: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:26.252851 26631 raft_consensus.cc:2932] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.259359 26293 consensus_queue.cc:237] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } }
I20240503 14:15:26.261265 26631 raft_consensus.cc:1261] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972 [term 1 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:26.261972 26949 consensus_queue.cc:1035] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:26.265324 26949 raft_consensus.cc:2932] T 4dee80b89ad7448e87bbd976b78a24d0 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dd1204c91174803920e3a83d494a1fe (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.265880 26631 raft_consensus.cc:2932] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dd1204c91174803920e3a83d494a1fe (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.270979 25745 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dee80b89ad7448e87bbd976b78a24d0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:26.271404 25759 catalog_manager.cc:5484] T 4dee80b89ad7448e87bbd976b78a24d0 P a680f58704cf4c979e241a706d51b972 reported cstate change: config changed from index 5 to 6, VOTER 6dd1204c91174803920e3a83d494a1fe (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.272485 26293 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1bc68ced13df43679e4fc4e8aef08f97"
dest_uuid: "0d5deb936bb744b392e3a28be25aa55c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49990
I20240503 14:15:26.272780 26293 raft_consensus.cc:595] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:26.276021 25879 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dee80b89ad7448e87bbd976b78a24d0 with delete_type TABLET_DATA_TOMBSTONED (TS 6dd1204c91174803920e3a83d494a1fe not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40334
I20240503 14:15:26.276914 27230 tablet_replica.cc:330] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe: stopping tablet replica
I20240503 14:15:26.277442 27230 raft_consensus.cc:2229] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:26.277897 27230 raft_consensus.cc:2258] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:26.279734 27230 ts_tablet_manager.cc:1875] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:26.285343 27230 ts_tablet_manager.cc:1888] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:15:26.285530 27230 log.cc:1196] T 4dee80b89ad7448e87bbd976b78a24d0 P 6dd1204c91174803920e3a83d494a1fe: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/4dee80b89ad7448e87bbd976b78a24d0
I20240503 14:15:26.286660 25744 catalog_manager.cc:4830] TS 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305): tablet 4dee80b89ad7448e87bbd976b78a24d0 (table rebalance_test_table_2 [id=71eedb8dc1b14d2eb0d7ce00ccc4423e]) successfully deleted
I20240503 14:15:26.290037 27124 raft_consensus.cc:1050] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c: attempting to promote NON_VOTER a680f58704cf4c979e241a706d51b972 to VOTER
I20240503 14:15:26.290699 27124 consensus_queue.cc:237] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } }
I20240503 14:15:26.292806 26631 raft_consensus.cc:1261] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.292830 25901 raft_consensus.cc:1261] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.293422 26949 consensus_queue.cc:1035] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.293772 27124 consensus_queue.cc:1035] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.296727 27123 raft_consensus.cc:2932] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.297588 26631 raft_consensus.cc:2932] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.297904 25901 raft_consensus.cc:2932] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.302218 25757 catalog_manager.cc:5484] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe reported cstate change: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.307689 26293 consensus_queue.cc:237] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } }
I20240503 14:15:26.309460 26631 raft_consensus.cc:1261] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972 [term 1 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:26.310040 27124 consensus_queue.cc:1035] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:26.312419 27123 raft_consensus.cc:2932] T 5eba7a96b98a4c5bbef5d3b386021dac P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dd1204c91174803920e3a83d494a1fe (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.313017 26631 raft_consensus.cc:2932] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6dd1204c91174803920e3a83d494a1fe (127.10.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.316596 25745 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5eba7a96b98a4c5bbef5d3b386021dac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:26.317191 25759 catalog_manager.cc:5484] T 5eba7a96b98a4c5bbef5d3b386021dac P a680f58704cf4c979e241a706d51b972 reported cstate change: config changed from index 5 to 6, VOTER 6dd1204c91174803920e3a83d494a1fe (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.321415 25879 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eba7a96b98a4c5bbef5d3b386021dac with delete_type TABLET_DATA_TOMBSTONED (TS 6dd1204c91174803920e3a83d494a1fe not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40334
I20240503 14:15:26.321920 27230 tablet_replica.cc:330] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe: stopping tablet replica
I20240503 14:15:26.322104 27230 raft_consensus.cc:2229] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:26.322371 27230 raft_consensus.cc:2258] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:26.323204 27230 ts_tablet_manager.cc:1875] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:26.327723 27230 ts_tablet_manager.cc:1888] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:15:26.327852 27230 log.cc:1196] T 5eba7a96b98a4c5bbef5d3b386021dac P 6dd1204c91174803920e3a83d494a1fe: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/5eba7a96b98a4c5bbef5d3b386021dac
I20240503 14:15:26.328629 25744 catalog_manager.cc:4830] TS 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305): tablet 5eba7a96b98a4c5bbef5d3b386021dac (table rebalance_test_table_1 [id=417923559eac4111af5209e6035bd69e]) successfully deleted
I20240503 14:15:26.374153 27156 raft_consensus.cc:1050] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe: attempting to promote NON_VOTER 171289531539427c84649caa8aa7f86e to VOTER
I20240503 14:15:26.374830 27156 consensus_queue.cc:237] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:26.376865 26763 raft_consensus.cc:1261] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e [term 1 LEARNER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.376860 26293 raft_consensus.cc:1261] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.377458 26812 consensus_queue.cc:1035] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.377768 27156 consensus_queue.cc:1035] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.380576 27125 raft_consensus.cc:2932] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.381342 26763 raft_consensus.cc:2932] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.381379 26293 raft_consensus.cc:2932] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.385912 25752 catalog_manager.cc:5484] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e reported cstate change: config changed from index 4 to 5, 171289531539427c84649caa8aa7f86e (127.10.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.391270 25901 consensus_queue.cc:237] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:26.393157 26763 raft_consensus.cc:1261] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e [term 1 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:26.393798 27125 consensus_queue.cc:1035] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:26.396235 26812 raft_consensus.cc:2932] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.396895 26763 raft_consensus.cc:2932] T cd66d059a2124fa985e0b12f6c9fa37e P 171289531539427c84649caa8aa7f86e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:26.400251 25744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd66d059a2124fa985e0b12f6c9fa37e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:26.402191 25759 catalog_manager.cc:5484] T cd66d059a2124fa985e0b12f6c9fa37e P 6dd1204c91174803920e3a83d494a1fe reported cstate change: config changed from index 5 to 6, VOTER 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:26.406095 26274 tablet_service.cc:1514] Processing DeleteTablet for tablet cd66d059a2124fa985e0b12f6c9fa37e with delete_type TABLET_DATA_TOMBSTONED (TS 0d5deb936bb744b392e3a28be25aa55c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49780
I20240503 14:15:26.407127 27242 tablet_replica.cc:330] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c: stopping tablet replica
I20240503 14:15:26.407578 27242 raft_consensus.cc:2229] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:26.408124 27242 raft_consensus.cc:2258] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:26.409842 27242 ts_tablet_manager.cc:1875] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:26.415001 27242 ts_tablet_manager.cc:1888] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:15:26.415163 27242 log.cc:1196] T cd66d059a2124fa985e0b12f6c9fa37e P 0d5deb936bb744b392e3a28be25aa55c: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/cd66d059a2124fa985e0b12f6c9fa37e
I20240503 14:15:26.416146 25745 catalog_manager.cc:4830] TS 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471): tablet cd66d059a2124fa985e0b12f6c9fa37e (table rebalance_test_table_1 [id=417923559eac4111af5209e6035bd69e]) successfully deleted
I20240503 14:15:26.503774 25901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "209eb1a4b190422ea47b12f96755f064"
dest_uuid: "6dd1204c91174803920e3a83d494a1fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40572
I20240503 14:15:26.504052 25901 raft_consensus.cc:595] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:26.549325 26293 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1bc68ced13df43679e4fc4e8aef08f97"
dest_uuid: "0d5deb936bb744b392e3a28be25aa55c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50026
I20240503 14:15:26.549525 26293 raft_consensus.cc:595] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:26.577080 25901 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "209eb1a4b190422ea47b12f96755f064"
dest_uuid: "6dd1204c91174803920e3a83d494a1fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40596
I20240503 14:15:26.577343 25901 raft_consensus.cc:595] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:26.593041 26812 raft_consensus.cc:1050] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe: attempting to promote NON_VOTER a680f58704cf4c979e241a706d51b972 to VOTER
I20240503 14:15:26.593755 26812 consensus_queue.cc:237] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } }
I20240503 14:15:26.596248 26293 raft_consensus.cc:1261] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.596359 26631 raft_consensus.cc:1261] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972 [term 1 LEARNER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:26.597092 27125 consensus_queue.cc:1035] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.597477 27156 consensus_queue.cc:1035] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:26.600551 27156 raft_consensus.cc:2932] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.604817 26631 raft_consensus.cc:2932] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.605901 26293 raft_consensus.cc:2932] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.606246 25759 catalog_manager.cc:5484] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe reported cstate change: config changed from index 4 to 5, a680f58704cf4c979e241a706d51b972 (127.10.145.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:15:26.613008 27156 consensus_queue.cc:1159] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a680f58704cf4c979e241a706d51b972" responder_term: 1 status { last_received { term: 1 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 5 } }
I20240503 14:15:26.613551 27124 raft_consensus.cc:979] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: : Instructing follower 6dd1204c91174803920e3a83d494a1fe to start an election
I20240503 14:15:26.613740 27124 raft_consensus.cc:1067] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Signalling peer 6dd1204c91174803920e3a83d494a1fe to start an election
I20240503 14:15:26.614399 25901 consensus_queue.cc:237] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } }
I20240503 14:15:26.616343 25901 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1bc68ced13df43679e4fc4e8aef08f97"
dest_uuid: "6dd1204c91174803920e3a83d494a1fe"
from {username='slave'} at 127.10.145.4:56763
I20240503 14:15:26.616544 25901 raft_consensus.cc:482] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:26.616694 25901 raft_consensus.cc:3037] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:26.616976 26631 raft_consensus.cc:1261] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972 [term 1 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:26.618094 27156 raft_consensus.cc:979] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: : Instructing follower 0d5deb936bb744b392e3a28be25aa55c to start an election
I20240503 14:15:26.618057 27125 consensus_queue.cc:1035] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:26.618312 27156 raft_consensus.cc:1067] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Signalling peer 0d5deb936bb744b392e3a28be25aa55c to start an election
I20240503 14:15:26.619309 26293 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "209eb1a4b190422ea47b12f96755f064"
dest_uuid: "0d5deb936bb744b392e3a28be25aa55c"
from {username='slave'} at 127.10.145.1:35585
I20240503 14:15:26.619555 26293 raft_consensus.cc:482] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:26.619482 25901 raft_consensus.cc:504] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:26.619706 26293 raft_consensus.cc:3037] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:26.620662 26292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc68ced13df43679e4fc4e8aef08f97" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "0d5deb936bb744b392e3a28be25aa55c"
I20240503 14:15:26.620865 26292 raft_consensus.cc:3032] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:26.620970 26292 raft_consensus.cc:729] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Running, Role: LEADER
I20240503 14:15:26.621373 26292 consensus_queue.cc:260] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:26.621766 26292 raft_consensus.cc:3037] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:26.622668 26293 raft_consensus.cc:504] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:26.623490 26293 leader_election.cc:288] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 2 election: Requested vote from peers 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305), 171289531539427c84649caa8aa7f86e (127.10.145.3:46613)
I20240503 14:15:26.624132 26763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209eb1a4b190422ea47b12f96755f064" candidate_uuid: "0d5deb936bb744b392e3a28be25aa55c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "171289531539427c84649caa8aa7f86e"
I20240503 14:15:26.624378 26292 raft_consensus.cc:2454] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 2.
I20240503 14:15:26.624497 26763 raft_consensus.cc:3037] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:26.625095 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209eb1a4b190422ea47b12f96755f064" candidate_uuid: "0d5deb936bb744b392e3a28be25aa55c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6dd1204c91174803920e3a83d494a1fe"
I20240503 14:15:26.625245 25835 leader_election.cc:302] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 6dd1204c91174803920e3a83d494a1fe; no voters:
I20240503 14:15:26.625396 25900 raft_consensus.cc:3032] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:26.625484 25900 raft_consensus.cc:729] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Running, Role: LEADER
I20240503 14:15:26.625723 27156 raft_consensus.cc:2781] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:26.625849 27156 raft_consensus.cc:686] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 2 LEADER]: Becoming Leader. State: Replica: 6dd1204c91174803920e3a83d494a1fe, State: Running, Role: LEADER
I20240503 14:15:26.626116 27156 consensus_queue.cc:237] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:26.626072 25900 consensus_queue.cc:260] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:26.626657 25900 raft_consensus.cc:3037] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:26.627403 26763 raft_consensus.cc:2454] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5deb936bb744b392e3a28be25aa55c in term 2.
I20240503 14:15:26.627439 27156 raft_consensus.cc:2932] T 44de804572554d56becf3387e1031e1b P 6dd1204c91174803920e3a83d494a1fe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.629739 26230 leader_election.cc:302] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5deb936bb744b392e3a28be25aa55c, 171289531539427c84649caa8aa7f86e; no voters:
I20240503 14:15:26.630020 25900 raft_consensus.cc:2454] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5deb936bb744b392e3a28be25aa55c in term 2.
I20240503 14:15:26.630125 27123 raft_consensus.cc:2781] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:26.630345 27123 raft_consensus.cc:686] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 2 LEADER]: Becoming Leader. State: Replica: 0d5deb936bb744b392e3a28be25aa55c, State: Running, Role: LEADER
I20240503 14:15:26.630672 27123 consensus_queue.cc:237] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:26.631729 25744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44de804572554d56becf3387e1031e1b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:26.632226 25901 leader_election.cc:288] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [CANDIDATE]: Term 2 election: Requested vote from peers 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471), 171289531539427c84649caa8aa7f86e (127.10.145.3:46613)
I20240503 14:15:26.633947 26763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc68ced13df43679e4fc4e8aef08f97" candidate_uuid: "6dd1204c91174803920e3a83d494a1fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "171289531539427c84649caa8aa7f86e"
I20240503 14:15:26.634218 26763 raft_consensus.cc:3037] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:26.634057 25752 catalog_manager.cc:5484] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c reported cstate change: term changed from 1 to 2, leader changed from 6dd1204c91174803920e3a83d494a1fe (127.10.145.1) to 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4). New cstate: current_term: 2 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:26.634080 26631 raft_consensus.cc:2932] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.636003 25757 catalog_manager.cc:5484] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe reported cstate change: term changed from 1 to 2, leader changed from 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4) to 6dd1204c91174803920e3a83d494a1fe (127.10.145.1). New cstate: current_term: 2 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:26.637270 26763 raft_consensus.cc:2454] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd1204c91174803920e3a83d494a1fe in term 2.
I20240503 14:15:26.641446 25757 catalog_manager.cc:5484] T 44de804572554d56becf3387e1031e1b P a680f58704cf4c979e241a706d51b972 reported cstate change: config changed from index 5 to 6, VOTER 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "a680f58704cf4c979e241a706d51b972" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 35219 } attrs { promote: false } } }
I20240503 14:15:26.644869 26274 tablet_service.cc:1514] Processing DeleteTablet for tablet 44de804572554d56becf3387e1031e1b with delete_type TABLET_DATA_TOMBSTONED (TS 0d5deb936bb744b392e3a28be25aa55c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49780
I20240503 14:15:26.645244 27242 tablet_replica.cc:330] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c: stopping tablet replica
I20240503 14:15:26.645481 27242 raft_consensus.cc:2229] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:26.645797 27242 raft_consensus.cc:2258] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:26.645773 26292 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1bc68ced13df43679e4fc4e8aef08f97"
dest_uuid: "0d5deb936bb744b392e3a28be25aa55c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50058
I20240503 14:15:26.645979 26292 raft_consensus.cc:595] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 14:15:26.646090 26292 raft_consensus.cc:603] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 14:15:26.646718 27242 ts_tablet_manager.cc:1875] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:26.651638 27242 ts_tablet_manager.cc:1888] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:15:26.651752 27242 log.cc:1196] T 44de804572554d56becf3387e1031e1b P 0d5deb936bb744b392e3a28be25aa55c: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/44de804572554d56becf3387e1031e1b
I20240503 14:15:26.652448 25745 catalog_manager.cc:4830] TS 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471): tablet 44de804572554d56becf3387e1031e1b (table rebalance_test_table_0 [id=938e33bb6bfb4575bf63a3643c9ee885]) successfully deleted
I20240503 14:15:27.031672 26763 raft_consensus.cc:1261] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [term 2 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:27.032459 26812 consensus_queue.cc:1035] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:27.037073 26292 raft_consensus.cc:1261] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 2 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:27.038051 26812 consensus_queue.cc:1035] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:27.039752 25901 consensus_queue.cc:237] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:27.041671 26763 raft_consensus.cc:1261] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [term 2 FOLLOWER]: Refusing update from remote peer 6dd1204c91174803920e3a83d494a1fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:15:27.042186 26812 consensus_queue.cc:1035] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:27.044389 27125 raft_consensus.cc:2932] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:27.045023 26763 raft_consensus.cc:2932] T 1bc68ced13df43679e4fc4e8aef08f97 P 171289531539427c84649caa8aa7f86e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:27.047320 25744 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1bc68ced13df43679e4fc4e8aef08f97 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:27.048588 25757 catalog_manager.cc:5484] T 1bc68ced13df43679e4fc4e8aef08f97 P 6dd1204c91174803920e3a83d494a1fe reported cstate change: config changed from index 5 to 7, VOTER 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4) evicted. New cstate: current_term: 2 leader_uuid: "6dd1204c91174803920e3a83d494a1fe" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:27.052639 26274 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc68ced13df43679e4fc4e8aef08f97 with delete_type TABLET_DATA_TOMBSTONED (TS 0d5deb936bb744b392e3a28be25aa55c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49780
I20240503 14:15:27.052976 27242 tablet_replica.cc:330] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: stopping tablet replica
I20240503 14:15:27.053179 27242 raft_consensus.cc:2229] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:27.053434 27242 raft_consensus.cc:2258] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:27.054322 27242 ts_tablet_manager.cc:1875] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:27.059005 27242 ts_tablet_manager.cc:1888] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:15:27.059139 27242 log.cc:1196] T 1bc68ced13df43679e4fc4e8aef08f97 P 0d5deb936bb744b392e3a28be25aa55c: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/1bc68ced13df43679e4fc4e8aef08f97
I20240503 14:15:27.059906 25745 catalog_manager.cc:4830] TS 0d5deb936bb744b392e3a28be25aa55c (127.10.145.4:37471): tablet 1bc68ced13df43679e4fc4e8aef08f97 (table rebalance_test_table_2 [id=71eedb8dc1b14d2eb0d7ce00ccc4423e]) successfully deleted
I20240503 14:15:27.114732 25901 raft_consensus.cc:1261] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 2 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:27.115296 27123 consensus_queue.cc:1035] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1204c91174803920e3a83d494a1fe" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 45305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:27.118780 26763 raft_consensus.cc:1261] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [term 2 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:27.119494 26949 consensus_queue.cc:1035] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:27.545959 26292 consensus_queue.cc:237] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } }
I20240503 14:15:27.547631 26763 raft_consensus.cc:1261] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [term 2 FOLLOWER]: Refusing update from remote peer 0d5deb936bb744b392e3a28be25aa55c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:15:27.548185 26949 consensus_queue.cc:1035] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:27.550869 27350 raft_consensus.cc:2932] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6dd1204c91174803920e3a83d494a1fe (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:27.551690 26763 raft_consensus.cc:2932] T 209eb1a4b190422ea47b12f96755f064 P 171289531539427c84649caa8aa7f86e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6dd1204c91174803920e3a83d494a1fe (127.10.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } } }
I20240503 14:15:27.555555 25745 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 209eb1a4b190422ea47b12f96755f064 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:27.556663 25752 catalog_manager.cc:5484] T 209eb1a4b190422ea47b12f96755f064 P 0d5deb936bb744b392e3a28be25aa55c reported cstate change: config changed from index 5 to 7, VOTER 6dd1204c91174803920e3a83d494a1fe (127.10.145.1) evicted. New cstate: current_term: 2 leader_uuid: "0d5deb936bb744b392e3a28be25aa55c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0d5deb936bb744b392e3a28be25aa55c" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 37471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171289531539427c84649caa8aa7f86e" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 46613 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:27.562752 25879 tablet_service.cc:1514] Processing DeleteTablet for tablet 209eb1a4b190422ea47b12f96755f064 with delete_type TABLET_DATA_TOMBSTONED (TS 6dd1204c91174803920e3a83d494a1fe not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40334
I20240503 14:15:27.563459 27376 tablet_replica.cc:330] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: stopping tablet replica
I20240503 14:15:27.563737 27376 raft_consensus.cc:2229] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:27.563987 27376 raft_consensus.cc:2258] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:27.564970 27376 ts_tablet_manager.cc:1875] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:27.570159 27376 ts_tablet_manager.cc:1888] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:15:27.570288 27376 log.cc:1196] T 209eb1a4b190422ea47b12f96755f064 P 6dd1204c91174803920e3a83d494a1fe: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/209eb1a4b190422ea47b12f96755f064
I20240503 14:15:27.570981 25744 catalog_manager.cc:4830] TS 6dd1204c91174803920e3a83d494a1fe (127.10.145.1:45305): tablet 209eb1a4b190422ea47b12f96755f064 (table rebalance_test_table_0 [id=938e33bb6bfb4575bf63a3643c9ee885]) successfully deleted
I20240503 14:15:27.648663 26274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:27.657588 25879 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:27.660068 26743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:27.661859 26612 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:27.823174 26274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:27.827566 25879 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:27.829655 26612 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:27.835961 26743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:27.987963 26743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:27.988051 26274 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:27.991010 26612 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:27.998662 25879 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:28.181207 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 25819
I20240503 14:15:28.204623 25941 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:28.359620 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 25819
W20240503 14:15:28.382190 26231 connection.cc:535] server connection from 127.10.145.1:35585 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:28.382614 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 26550
I20240503 14:15:28.388949 26672 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:28.496996 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 26550
I20240503 14:15:28.515242 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 26681
I20240503 14:15:28.522032 26802 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:28.630985 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 26681
W20240503 14:15:28.648348 25747 connection.cc:535] server connection from 127.10.145.3:38677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:28.648468 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 26212
I20240503 14:15:28.654881 26334 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:28.782647 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 26212
I20240503 14:15:28.803354 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 25729
I20240503 14:15:28.804576 25790 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:28.939034 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 25729
2024-05-03T14:15:28Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 (17256 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx6_rf7_t7_l_3_4_3
I20240503 14:15:28.992429 10820 test_util.cc:268] Using random seed: 1510900856
I20240503 14:15:28.994799 10820 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:15:28.994910 10820 ts_itest-base.cc:113] --------------
I20240503 14:15:28.994959 10820 ts_itest-base.cc:114] 10 tablet servers
I20240503 14:15:28.995021 10820 ts_itest-base.cc:115] 7 replicas per TS
I20240503 14:15:28.995066 10820 ts_itest-base.cc:116] --------------
2024-05-03T14:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:15:29Z Disabled control of system clock
I20240503 14:15:29.049319 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:42867
--webserver_interface=127.10.145.62
--webserver_port=0
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.145.62:42867
--location_mapping_cmd=/tmp/dist-test-taskFZBpI6/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 14:15:29.405184 27490 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 14:15:29.405673 27490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:29.405772 27490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:29.405880 27490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:29.413972 27490 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:29.414144 27490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:15:29.414252 27490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:29.414312 27490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:15:29.414400 27490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:15:29.424455 27490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskFZBpI6/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--ipki_ca_key_size=768
--master_addresses=127.10.145.62:42867
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.145.62:42867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:29.425979 27490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:29.427872 27490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:29.439251 27495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:29.439587 27498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:29.439893 27496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:29.440160 27490 server_base.cc:1034] running on GCE node
I20240503 14:15:29.440972 27490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:29.442623 27490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:29.443845 27490 hybrid_clock.cc:648] HybridClock initialized: now 1714745729443781 us; error 44 us; skew 500 ppm
I20240503 14:15:29.444205 27490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:29.446617 27490 webserver.cc:460] Webserver started at http://127.10.145.62:34749/ using document root <none> and password file <none>
I20240503 14:15:29.447463 27490 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:29.447593 27490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:29.447911 27490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:29.450789 27490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9c3174ded162464fbb5090cdd287cb0c"
format_stamp: "Formatted at 2024-05-03 14:15:29 on dist-test-slave-v549"
I20240503 14:15:29.451649 27490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9c3174ded162464fbb5090cdd287cb0c"
format_stamp: "Formatted at 2024-05-03 14:15:29 on dist-test-slave-v549"
I20240503 14:15:29.457525 27490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 14:15:29.461493 27504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:29.463080 27490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:29.463287 27490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9c3174ded162464fbb5090cdd287cb0c"
format_stamp: "Formatted at 2024-05-03 14:15:29 on dist-test-slave-v549"
I20240503 14:15:29.463480 27490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:29.489367 27490 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:29.490316 27490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:29.490628 27490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:29.508915 27490 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.62:42867
I20240503 14:15:29.508936 27555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.62:42867 every 8 connection(s)
I20240503 14:15:29.510496 27490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:15:29.514617 27556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:29.519739 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 27490
I20240503 14:15:29.519969 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:15:29.525732 27556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c: Bootstrap starting.
I20240503 14:15:29.530081 27556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:29.531206 27556 log.cc:824] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:29.534263 27556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c: No bootstrap required, opened a new log
I20240503 14:15:29.541321 27556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3174ded162464fbb5090cdd287cb0c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42867 } }
I20240503 14:15:29.541754 27556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:29.541857 27556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c3174ded162464fbb5090cdd287cb0c, State: Initialized, Role: FOLLOWER
I20240503 14:15:29.542474 27556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3174ded162464fbb5090cdd287cb0c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42867 } }
I20240503 14:15:29.542701 27556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:29.542822 27556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:29.542979 27556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:29.545236 27556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3174ded162464fbb5090cdd287cb0c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42867 } }
I20240503 14:15:29.545912 27556 leader_election.cc:302] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c3174ded162464fbb5090cdd287cb0c; no voters:
I20240503 14:15:29.546545 27556 leader_election.cc:288] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:29.546682 27561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:29.547369 27561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [term 1 LEADER]: Becoming Leader. State: Replica: 9c3174ded162464fbb5090cdd287cb0c, State: Running, Role: LEADER
I20240503 14:15:29.547919 27561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3174ded162464fbb5090cdd287cb0c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42867 } }
I20240503 14:15:29.548739 27556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:15:29.550861 27563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c3174ded162464fbb5090cdd287cb0c. Latest consensus state: current_term: 1 leader_uuid: "9c3174ded162464fbb5090cdd287cb0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3174ded162464fbb5090cdd287cb0c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42867 } } }
I20240503 14:15:29.551196 27563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [sys.catalog]: This master's current role is: LEADER
I20240503 14:15:29.554872 27562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c3174ded162464fbb5090cdd287cb0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3174ded162464fbb5090cdd287cb0c" member_type: VOTER last_known_addr { host: "127.10.145.62" port: 42867 } } }
I20240503 14:15:29.555133 27562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c [sys.catalog]: This master's current role is: LEADER
I20240503 14:15:29.558728 27567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:15:29.563827 27567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:15:29.573678 27567 catalog_manager.cc:1347] Generated new cluster ID: 2e50825b9bc44927be4aeacbffe4fdd3
I20240503 14:15:29.573802 27567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:15:29.593204 27567 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:15:29.594656 27567 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:15:29.602882 27567 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9c3174ded162464fbb5090cdd287cb0c: Generated new TSK 0
I20240503 14:15:29.603861 27567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:15:29.620081 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.1:0
--local_ip_for_outbound_sockets=127.10.145.1
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:15:29.981817 27580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:29.982173 27580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:29.982293 27580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:29.990062 27580 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:29.990202 27580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:29.990355 27580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.1
I20240503 14:15:29.999917 27580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.145.1
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.1
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:30.001456 27580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:30.003351 27580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:30.015934 27585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:30.016259 27588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:30.016619 27586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:30.016891 27580 server_base.cc:1034] running on GCE node
I20240503 14:15:30.017596 27580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:30.018632 27580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:30.019855 27580 hybrid_clock.cc:648] HybridClock initialized: now 1714745730019804 us; error 42 us; skew 500 ppm
I20240503 14:15:30.020226 27580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:30.022691 27580 webserver.cc:460] Webserver started at http://127.10.145.1:38247/ using document root <none> and password file <none>
I20240503 14:15:30.023480 27580 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:30.023593 27580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:30.023916 27580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:30.026856 27580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7046cbd82d634976b53fe0cef5788f18"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-v549"
I20240503 14:15:30.027735 27580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7046cbd82d634976b53fe0cef5788f18"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-v549"
I20240503 14:15:30.033758 27580 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 14:15:30.037835 27594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.039551 27580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:30.039765 27580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7046cbd82d634976b53fe0cef5788f18"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-v549"
I20240503 14:15:30.039994 27580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:30.054093 27580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:30.055088 27580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:30.055402 27580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:30.056439 27580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:30.058256 27580 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:30.058357 27580 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.058465 27580 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:30.058537 27580 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.089432 27580 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.1:35261
I20240503 14:15:30.089460 27706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.1:35261 every 8 connection(s)
I20240503 14:15:30.090847 27580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:15:30.096449 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 27580
I20240503 14:15:30.096693 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:15:30.100433 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.2:0
--local_ip_for_outbound_sockets=127.10.145.2
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:30.106151 27707 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:30.106541 27707 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:30.107642 27707 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:30.154270 27521 ts_manager.cc:194] Registered new tserver with Master: 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:15:30.156178 27521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.1:56797
W20240503 14:15:30.460281 27711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:30.460624 27711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:30.460758 27711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:30.468550 27711 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:30.468724 27711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:30.468891 27711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.2
I20240503 14:15:30.478308 27711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.145.2
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.2
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:30.479847 27711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:30.481776 27711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:30.495074 27717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:30.495426 27720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:30.495777 27718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:30.495941 27711 server_base.cc:1034] running on GCE node
I20240503 14:15:30.496636 27711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:30.497709 27711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:30.498925 27711 hybrid_clock.cc:648] HybridClock initialized: now 1714745730498871 us; error 44 us; skew 500 ppm
I20240503 14:15:30.499291 27711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:30.501773 27711 webserver.cc:460] Webserver started at http://127.10.145.2:39359/ using document root <none> and password file <none>
I20240503 14:15:30.502614 27711 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:30.502743 27711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:30.503069 27711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:30.505950 27711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "74318b6270b64efaa79867167fc08bbb"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-v549"
I20240503 14:15:30.506829 27711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "74318b6270b64efaa79867167fc08bbb"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-v549"
I20240503 14:15:30.513705 27711 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20240503 14:15:30.518519 27726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.520255 27711 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20240503 14:15:30.520450 27711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "74318b6270b64efaa79867167fc08bbb"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-v549"
I20240503 14:15:30.520704 27711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:30.575500 27711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:30.576465 27711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:30.576736 27711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:30.577800 27711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:30.579464 27711 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:30.579560 27711 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.579651 27711 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:30.579705 27711 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.611063 27711 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.2:38859
I20240503 14:15:30.611111 27838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.2:38859 every 8 connection(s)
I20240503 14:15:30.612463 27711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:15:30.614089 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 27711
I20240503 14:15:30.614271 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:15:30.617951 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.3:0
--local_ip_for_outbound_sockets=127.10.145.3
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:30.628050 27839 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:30.628432 27839 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:30.629583 27839 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:30.677971 27521 ts_manager.cc:194] Registered new tserver with Master: 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859)
I20240503 14:15:30.679287 27521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.2:54033
W20240503 14:15:30.978255 27843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:30.978574 27843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:30.978664 27843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:30.986198 27843 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:30.986335 27843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:30.986456 27843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.3
I20240503 14:15:30.995730 27843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.145.3
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.3
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:30.997203 27843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:30.999187 27843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:31.011687 27849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:31.012013 27850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:31.012498 27852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:31.012938 27843 server_base.cc:1034] running on GCE node
I20240503 14:15:31.013650 27843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:31.014667 27843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:31.015906 27843 hybrid_clock.cc:648] HybridClock initialized: now 1714745731015845 us; error 49 us; skew 500 ppm
I20240503 14:15:31.016285 27843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:31.018795 27843 webserver.cc:460] Webserver started at http://127.10.145.3:34571/ using document root <none> and password file <none>
I20240503 14:15:31.019562 27843 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:31.019676 27843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:31.020009 27843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:31.022869 27843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a242f99f54b4494893064cfe3030b4ad"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-v549"
I20240503 14:15:31.023730 27843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a242f99f54b4494893064cfe3030b4ad"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-v549"
I20240503 14:15:31.029500 27843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:15:31.033111 27858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.034691 27843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:31.034852 27843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a242f99f54b4494893064cfe3030b4ad"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-v549"
I20240503 14:15:31.035053 27843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:31.054172 27843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:31.055109 27843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:31.055383 27843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:31.056402 27843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:31.058038 27843 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:31.058130 27843 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.058243 27843 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:31.058362 27843 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 14:15:31.089213 27843 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.3:36119
I20240503 14:15:31.089251 27970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.3:36119 every 8 connection(s)
I20240503 14:15:31.091039 27843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:15:31.095057 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 27843
I20240503 14:15:31.095250 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:15:31.098608 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.4:0
--local_ip_for_outbound_sockets=127.10.145.4
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:31.104974 27971 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:31.105357 27971 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:31.106268 27971 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:31.152252 27521 ts_manager.cc:194] Registered new tserver with Master: a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:31.153573 27521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.3:50881
I20240503 14:15:31.160238 27707 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
W20240503 14:15:31.460323 27974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:31.460662 27974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:31.460754 27974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:31.468638 27974 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:31.468787 27974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:31.468919 27974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.4
I20240503 14:15:31.479144 27974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.145.4
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.4
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:31.480721 27974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:31.482769 27974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:31.496698 27980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:31.497014 27981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:31.497365 27983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:31.497761 27974 server_base.cc:1034] running on GCE node
I20240503 14:15:31.498405 27974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:31.499475 27974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:31.500695 27974 hybrid_clock.cc:648] HybridClock initialized: now 1714745731500629 us; error 64 us; skew 500 ppm
I20240503 14:15:31.501044 27974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:31.508248 27974 webserver.cc:460] Webserver started at http://127.10.145.4:38077/ using document root <none> and password file <none>
I20240503 14:15:31.509168 27974 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:31.509330 27974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:31.509687 27974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:31.512518 27974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "078f8eff84c94dde9705ad1330922a11"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-v549"
I20240503 14:15:31.513464 27974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "078f8eff84c94dde9705ad1330922a11"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-v549"
I20240503 14:15:31.519196 27974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:15:31.523088 27989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.524653 27974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20240503 14:15:31.524817 27974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "078f8eff84c94dde9705ad1330922a11"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-v549"
I20240503 14:15:31.525022 27974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:31.544795 27974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:31.545861 27974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:31.546149 27974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:31.547401 27974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:31.549126 27974 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:31.549221 27974 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.549371 27974 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:31.549449 27974 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.580562 27974 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.4:45951
I20240503 14:15:31.580613 28101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.4:45951 every 8 connection(s)
I20240503 14:15:31.582031 27974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:15:31.584988 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 27974
I20240503 14:15:31.585194 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:15:31.589073 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.5:0
--local_ip_for_outbound_sockets=127.10.145.5
--webserver_interface=127.10.145.5
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:31.597913 28102 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:31.598294 28102 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:31.599347 28102 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:31.661016 27521 ts_manager.cc:194] Registered new tserver with Master: 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:15:31.662070 27521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.4:36657
I20240503 14:15:31.682349 27839 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
W20240503 14:15:31.968585 28105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:31.968940 28105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:31.969084 28105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:31.976824 28105 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:31.976997 28105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:31.977165 28105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.5
I20240503 14:15:31.987048 28105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.10.145.5
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.5
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:31.988528 28105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:31.990473 28105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:32.003305 28111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:32.003444 28112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:32.003844 28114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:32.004194 28105 server_base.cc:1034] running on GCE node
I20240503 14:15:32.004853 28105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:32.005926 28105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:32.007133 28105 hybrid_clock.cc:648] HybridClock initialized: now 1714745732007074 us; error 56 us; skew 500 ppm
I20240503 14:15:32.007472 28105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:32.009903 28105 webserver.cc:460] Webserver started at http://127.10.145.5:37221/ using document root <none> and password file <none>
I20240503 14:15:32.010710 28105 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:32.010833 28105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:32.011162 28105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:32.014102 28105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-v549"
I20240503 14:15:32.014981 28105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-v549"
I20240503 14:15:32.020747 28105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:15:32.024714 28120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.026305 28105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:32.026497 28105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-v549"
I20240503 14:15:32.026727 28105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:32.056293 28105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:32.057340 28105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:32.057641 28105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:32.058770 28105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:32.060449 28105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:32.060554 28105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.060674 28105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:32.060742 28105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.095216 28105 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.5:35605
I20240503 14:15:32.095274 28232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.5:35605 every 8 connection(s)
I20240503 14:15:32.096798 28105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:15:32.099673 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 28105
I20240503 14:15:32.099860 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:15:32.103924 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.6:0
--local_ip_for_outbound_sockets=127.10.145.6
--webserver_interface=127.10.145.6
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:32.111375 28233 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:32.111768 28233 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:32.112854 28233 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:32.156615 27971 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
I20240503 14:15:32.162204 27521 ts_manager.cc:194] Registered new tserver with Master: c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605)
I20240503 14:15:32.163236 27521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.5:37127
W20240503 14:15:32.477658 28237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:32.478000 28237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:32.478096 28237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:32.485879 28237 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:32.486025 28237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:32.486156 28237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.6
I20240503 14:15:32.497066 28237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.10.145.6
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.6
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:32.498606 28237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:32.500566 28237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:32.513559 28244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:32.513845 28243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:32.514350 28246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:32.514585 28237 server_base.cc:1034] running on GCE node
I20240503 14:15:32.515367 28237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:32.516584 28237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:32.517814 28237 hybrid_clock.cc:648] HybridClock initialized: now 1714745732517748 us; error 51 us; skew 500 ppm
I20240503 14:15:32.518224 28237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:32.520570 28237 webserver.cc:460] Webserver started at http://127.10.145.6:42251/ using document root <none> and password file <none>
I20240503 14:15:32.521373 28237 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:32.521479 28237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:32.521773 28237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:32.524566 28237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "acb93936f2f14e628eecb05d7d1f4d91"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-v549"
I20240503 14:15:32.525416 28237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "acb93936f2f14e628eecb05d7d1f4d91"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-v549"
I20240503 14:15:32.530927 28237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240503 14:15:32.534530 28252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.536031 28237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:32.536211 28237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "acb93936f2f14e628eecb05d7d1f4d91"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-v549"
I20240503 14:15:32.536418 28237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:32.560966 28237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:32.561975 28237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:32.562265 28237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:32.563310 28237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:32.564950 28237 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:32.565057 28237 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.565171 28237 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:32.565237 28237 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.595522 28237 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.6:36073
I20240503 14:15:32.595614 28364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.6:36073 every 8 connection(s)
I20240503 14:15:32.596868 28237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:15:32.600189 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 28237
I20240503 14:15:32.600409 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:15:32.604249 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.7:0
--local_ip_for_outbound_sockets=127.10.145.7
--webserver_interface=127.10.145.7
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:32.610960 28365 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:32.611338 28365 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:32.612470 28365 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:32.659241 27520 ts_manager.cc:194] Registered new tserver with Master: acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:15:32.660320 27520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.6:37975
I20240503 14:15:32.665733 28102 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
W20240503 14:15:32.961935 28368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:32.962298 28368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:32.962433 28368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:32.970141 28368 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:32.970301 28368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:32.970465 28368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.7
I20240503 14:15:32.979730 28368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.10.145.7
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.7
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:32.981212 28368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:32.983110 28368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:32.995795 28374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:32.996075 28377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:32.996400 28375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:32.996853 28368 server_base.cc:1034] running on GCE node
I20240503 14:15:32.997551 28368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:32.998634 28368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:32.999859 28368 hybrid_clock.cc:648] HybridClock initialized: now 1714745732999810 us; error 49 us; skew 500 ppm
I20240503 14:15:33.000290 28368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:33.002753 28368 webserver.cc:460] Webserver started at http://127.10.145.7:34721/ using document root <none> and password file <none>
I20240503 14:15:33.003598 28368 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:33.003724 28368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:33.004042 28368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:33.007017 28368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "9231e715a4e342a197d6c5456ba6b399"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-v549"
I20240503 14:15:33.007885 28368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "9231e715a4e342a197d6c5456ba6b399"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-v549"
I20240503 14:15:33.013656 28368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240503 14:15:33.017490 28383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:33.018980 28368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:33.019168 28368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9231e715a4e342a197d6c5456ba6b399"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-v549"
I20240503 14:15:33.019382 28368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:33.061738 28368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:33.062724 28368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:33.063026 28368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:33.064101 28368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:33.065761 28368 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:33.065862 28368 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:33.065974 28368 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:33.066044 28368 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:33.108206 28368 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.7:43675
I20240503 14:15:33.108266 28495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.7:43675 every 8 connection(s)
I20240503 14:15:33.109716 28368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:15:33.112663 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 28368
I20240503 14:15:33.112859 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:15:33.116791 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.8:0
--local_ip_for_outbound_sockets=127.10.145.8
--webserver_interface=127.10.145.8
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:33.124567 28496 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:33.124987 28496 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:33.126173 28496 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:33.166366 28233 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
I20240503 14:15:33.217471 27520 ts_manager.cc:194] Registered new tserver with Master: 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:33.218513 27520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.7:39015
W20240503 14:15:33.494554 28500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:33.494930 28500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:33.495072 28500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:33.504099 28500 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:33.504266 28500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:33.504433 28500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.8
I20240503 14:15:33.514117 28500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.10.145.8
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.8
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:33.515569 28500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:33.517526 28500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:33.530402 28506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:33.531050 28507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:33.530649 28509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:33.531451 28500 server_base.cc:1034] running on GCE node
I20240503 14:15:33.532148 28500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:33.533178 28500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:33.534402 28500 hybrid_clock.cc:648] HybridClock initialized: now 1714745733534295 us; error 93 us; skew 500 ppm
I20240503 14:15:33.534762 28500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:33.537153 28500 webserver.cc:460] Webserver started at http://127.10.145.8:34263/ using document root <none> and password file <none>
I20240503 14:15:33.538054 28500 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:33.538182 28500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:33.538548 28500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:33.541606 28500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e938d875af584f1f88a2d4b50c589268"
format_stamp: "Formatted at 2024-05-03 14:15:33 on dist-test-slave-v549"
I20240503 14:15:33.542477 28500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e938d875af584f1f88a2d4b50c589268"
format_stamp: "Formatted at 2024-05-03 14:15:33 on dist-test-slave-v549"
I20240503 14:15:33.548236 28500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:15:33.552145 28515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:33.553846 28500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 14:15:33.554034 28500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e938d875af584f1f88a2d4b50c589268"
format_stamp: "Formatted at 2024-05-03 14:15:33 on dist-test-slave-v549"
I20240503 14:15:33.554256 28500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:33.593878 28500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:33.594898 28500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:33.595219 28500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:33.596325 28500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:33.598045 28500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:33.598142 28500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:33.598258 28500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:33.598325 28500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:33.629927 28500 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.8:44583
I20240503 14:15:33.630002 28627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.8:44583 every 8 connection(s)
I20240503 14:15:33.631475 28500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:15:33.641268 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 28500
I20240503 14:15:33.641508 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:15:33.646574 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.9:0
--local_ip_for_outbound_sockets=127.10.145.9
--webserver_interface=127.10.145.9
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:33.647235 28628 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:33.647607 28628 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:33.648842 28628 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:33.663326 28365 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
I20240503 14:15:33.698920 27520 ts_manager.cc:194] Registered new tserver with Master: e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583)
I20240503 14:15:33.700256 27520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.8:51013
W20240503 14:15:34.002835 28633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:34.003157 28633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:34.003247 28633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:34.010845 28633 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:34.010989 28633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:34.011108 28633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.9
I20240503 14:15:34.020843 28633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.9:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.10.145.9
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.9
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:34.022404 28633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:34.024273 28633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:34.037468 28638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:34.037590 28641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:34.038022 28639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:34.038348 28633 server_base.cc:1034] running on GCE node
I20240503 14:15:34.039041 28633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:34.040112 28633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:34.041337 28633 hybrid_clock.cc:648] HybridClock initialized: now 1714745734041278 us; error 54 us; skew 500 ppm
I20240503 14:15:34.041728 28633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:34.044174 28633 webserver.cc:460] Webserver started at http://127.10.145.9:46745/ using document root <none> and password file <none>
I20240503 14:15:34.045042 28633 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:34.045150 28633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:34.045589 28633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:34.048718 28633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "54dab61f1a204c0da4f56f98b3fd2690"
format_stamp: "Formatted at 2024-05-03 14:15:34 on dist-test-slave-v549"
I20240503 14:15:34.049610 28633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "54dab61f1a204c0da4f56f98b3fd2690"
format_stamp: "Formatted at 2024-05-03 14:15:34 on dist-test-slave-v549"
I20240503 14:15:34.055287 28633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:15:34.059177 28647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:34.060889 28633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:34.061103 28633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "54dab61f1a204c0da4f56f98b3fd2690"
format_stamp: "Formatted at 2024-05-03 14:15:34 on dist-test-slave-v549"
I20240503 14:15:34.061399 28633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:34.088654 28633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:34.090062 28633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:34.090430 28633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:34.091807 28633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:34.093746 28633 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:34.093850 28633 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:34.093955 28633 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:34.094017 28633 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:34.125312 28633 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.9:39057
I20240503 14:15:34.125363 28759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.9:39057 every 8 connection(s)
I20240503 14:15:34.126729 28633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 14:15:34.134552 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 28633
I20240503 14:15:34.134753 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240503 14:15:34.138346 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.10:0
--local_ip_for_outbound_sockets=127.10.145.10
--webserver_interface=127.10.145.10
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:34.140579 28760 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:34.140993 28760 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:34.142077 28760 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:34.188869 27520 ts_manager.cc:194] Registered new tserver with Master: 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057)
I20240503 14:15:34.189880 27520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.9:40359
I20240503 14:15:34.221422 28496 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
W20240503 14:15:34.495064 28763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:34.495378 28763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:34.495510 28763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:34.504501 28763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:34.504632 28763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:34.504784 28763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.10
I20240503 14:15:34.514235 28763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.10:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.10.145.10
--webserver_port=0
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.10
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:34.515640 28763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:34.517560 28763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:34.530099 28772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:34.530380 28769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:34.530748 28770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:34.531230 28763 server_base.cc:1034] running on GCE node
I20240503 14:15:34.531836 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:34.532806 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:34.534013 28763 hybrid_clock.cc:648] HybridClock initialized: now 1714745734533947 us; error 55 us; skew 500 ppm
I20240503 14:15:34.534345 28763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:34.536612 28763 webserver.cc:460] Webserver started at http://127.10.145.10:44195/ using document root <none> and password file <none>
I20240503 14:15:34.537429 28763 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:34.537539 28763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:34.537832 28763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:34.540656 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data/instance:
uuid: "01173ed91fd341e3a4a4599118167e52"
format_stamp: "Formatted at 2024-05-03 14:15:34 on dist-test-slave-v549"
I20240503 14:15:34.541512 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal/instance:
uuid: "01173ed91fd341e3a4a4599118167e52"
format_stamp: "Formatted at 2024-05-03 14:15:34 on dist-test-slave-v549"
I20240503 14:15:34.546917 28763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:15:34.550522 28778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:34.551931 28763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240503 14:15:34.552086 28763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "01173ed91fd341e3a4a4599118167e52"
format_stamp: "Formatted at 2024-05-03 14:15:34 on dist-test-slave-v549"
I20240503 14:15:34.552287 28763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:34.568274 28763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:34.569222 28763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:34.569506 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:34.570509 28763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:34.572037 28763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:34.572124 28763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:34.572207 28763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:34.572258 28763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:34.602213 28763 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.10:40989
I20240503 14:15:34.602263 28890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.10:40989 every 8 connection(s)
I20240503 14:15:34.603575 28763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240503 14:15:34.611573 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 28763
I20240503 14:15:34.611867 10820 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal/instance
I20240503 14:15:34.617811 28891 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:34.618098 28891 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:34.618933 28891 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:34.665222 27520 ts_manager.cc:194] Registered new tserver with Master: 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989)
I20240503 14:15:34.666312 27520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.10:37953
I20240503 14:15:34.668918 10820 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240503 14:15:34.688271 10820 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240503 14:15:34.688436 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 28500
I20240503 14:15:34.697556 28623 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:34.778671 28628 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
I20240503 14:15:34.780071 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 28500
W20240503 14:15:34.793642 27505 connection.cc:535] server connection from 127.10.145.8:51013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:34.793797 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 28633
I20240503 14:15:34.802763 28755 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:34.887125 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 28633
I20240503 14:15:34.901008 10820 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu pid 28763
I20240503 14:15:34.910140 28886 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:34.991678 10820 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu with pid 28763
I20240503 14:15:38.757967 27519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51414:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:38.766041 27519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 14:15:38.816320 28429 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae969579ee545c9838d3c26e4abd66f (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.817967 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae969579ee545c9838d3c26e4abd66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.818074 28430 tablet_service.cc:1467] Processing CreateTablet for tablet add941da26824cdda7489b5da990e879 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.818557 28430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add941da26824cdda7489b5da990e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.814987 28431 tablet_service.cc:1467] Processing CreateTablet for tablet 12dae8a392374b91be378ec431838686 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.820889 28431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dae8a392374b91be378ec431838686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.843916 28167 tablet_service.cc:1467] Processing CreateTablet for tablet add941da26824cdda7489b5da990e879 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.845909 28299 tablet_service.cc:1467] Processing CreateTablet for tablet add941da26824cdda7489b5da990e879 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.846099 28300 tablet_service.cc:1467] Processing CreateTablet for tablet 12dae8a392374b91be378ec431838686 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.847736 28300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dae8a392374b91be378ec431838686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.849030 28166 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae969579ee545c9838d3c26e4abd66f (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.849972 28166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae969579ee545c9838d3c26e4abd66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.852213 28037 tablet_service.cc:1467] Processing CreateTablet for tablet 12dae8a392374b91be378ec431838686 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.853678 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dae8a392374b91be378ec431838686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.853900 28036 tablet_service.cc:1467] Processing CreateTablet for tablet add941da26824cdda7489b5da990e879 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.854439 28036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add941da26824cdda7489b5da990e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.856276 28035 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae969579ee545c9838d3c26e4abd66f (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.856819 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae969579ee545c9838d3c26e4abd66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.859026 27774 tablet_service.cc:1467] Processing CreateTablet for tablet 12dae8a392374b91be378ec431838686 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.860487 27774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dae8a392374b91be378ec431838686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.858727 27772 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae969579ee545c9838d3c26e4abd66f (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.861886 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae969579ee545c9838d3c26e4abd66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.865957 27905 tablet_service.cc:1467] Processing CreateTablet for tablet add941da26824cdda7489b5da990e879 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.866771 27906 tablet_service.cc:1467] Processing CreateTablet for tablet 12dae8a392374b91be378ec431838686 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.867789 27906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dae8a392374b91be378ec431838686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.871706 27905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add941da26824cdda7489b5da990e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.846252 28298 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae969579ee545c9838d3c26e4abd66f (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.873807 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae969579ee545c9838d3c26e4abd66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.866286 27904 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae969579ee545c9838d3c26e4abd66f (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.877948 27904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae969579ee545c9838d3c26e4abd66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.841449 28168 tablet_service.cc:1467] Processing CreateTablet for tablet 12dae8a392374b91be378ec431838686 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.882896 28168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dae8a392374b91be378ec431838686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.883679 28167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add941da26824cdda7489b5da990e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.858592 27773 tablet_service.cc:1467] Processing CreateTablet for tablet add941da26824cdda7489b5da990e879 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.884632 28299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add941da26824cdda7489b5da990e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.886583 27773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add941da26824cdda7489b5da990e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.897997 28925 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:38.903874 28924 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:38.904844 28925 tablet_bootstrap.cc:654] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.906446 28925 log.cc:824] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:38.908380 28924 tablet_bootstrap.cc:654] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.909893 28924 log.cc:824] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:38.917482 28925 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:38.917922 28925 ts_tablet_manager.cc:1367] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20240503 14:15:38.919260 28924 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:38.919564 28924 ts_tablet_manager.cc:1367] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20240503 14:15:38.928537 28924 raft_consensus.cc:348] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.928539 28925 raft_consensus.cc:348] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.929152 28924 raft_consensus.cc:374] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.929262 28924 raft_consensus.cc:729] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.929509 28925 raft_consensus.cc:374] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.929701 28925 raft_consensus.cc:729] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.930266 28924 consensus_queue.cc:260] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.930789 28925 consensus_queue.cc:260] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.939003 28924 ts_tablet_manager.cc:1398] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.019s user 0.014s sys 0.005s
I20240503 14:15:38.939522 28924 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:38.941486 28924 tablet_bootstrap.cc:654] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.941996 28925 ts_tablet_manager.cc:1398] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.024s user 0.010s sys 0.008s
I20240503 14:15:38.944278 28925 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:38.946439 28925 tablet_bootstrap.cc:654] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.950022 28924 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:38.950219 28924 ts_tablet_manager.cc:1367] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 14:15:38.951376 28924 raft_consensus.cc:348] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.951697 28924 raft_consensus.cc:374] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.951779 28924 raft_consensus.cc:729] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.952088 28924 consensus_queue.cc:260] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.952839 28924 ts_tablet_manager.cc:1398] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:15:38.953140 28924 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:38.955093 28924 tablet_bootstrap.cc:654] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.961253 28925 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:38.961585 28925 ts_tablet_manager.cc:1367] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.002s
I20240503 14:15:38.962980 28925 raft_consensus.cc:348] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.963403 28925 raft_consensus.cc:374] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.963541 28925 raft_consensus.cc:729] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.963910 28925 consensus_queue.cc:260] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.964761 28925 ts_tablet_manager.cc:1398] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:38.965152 28925 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:38.967144 28925 tablet_bootstrap.cc:654] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.970269 28924 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:38.970497 28924 ts_tablet_manager.cc:1367] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20240503 14:15:38.971442 28924 raft_consensus.cc:348] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.971807 28924 raft_consensus.cc:374] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.971879 28924 raft_consensus.cc:729] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.972199 28924 consensus_queue.cc:260] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.972983 28924 ts_tablet_manager.cc:1398] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:15:38.979759 28935 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:38.983855 28934 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:38.984519 28935 tablet_bootstrap.cc:654] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.985723 28923 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:38.986086 28935 log.cc:824] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:38.989938 28929 raft_consensus.cc:482] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:38.990243 28929 raft_consensus.cc:504] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:38.996516 28935 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:38.996872 28935 ts_tablet_manager.cc:1367] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240503 14:15:38.999493 27641 tablet_service.cc:1467] Processing CreateTablet for tablet add941da26824cdda7489b5da990e879 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.000217 28934 tablet_bootstrap.cc:654] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.001226 27641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add941da26824cdda7489b5da990e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.001612 28934 log.cc:824] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:39.001636 27642 tablet_service.cc:1467] Processing CreateTablet for tablet 12dae8a392374b91be378ec431838686 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.002559 27640 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae969579ee545c9838d3c26e4abd66f (DEFAULT_TABLE table=rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.002950 27642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dae8a392374b91be378ec431838686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.003515 27640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae969579ee545c9838d3c26e4abd66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.003610 28929 leader_election.cc:288] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859)
I20240503 14:15:39.006069 28937 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:39.006688 28923 tablet_bootstrap.cc:654] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.008158 28923 log.cc:824] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:39.029687 28935 raft_consensus.cc:348] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.031628 28935 raft_consensus.cc:374] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.033089 28937 tablet_bootstrap.cc:654] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.034725 28937 log.cc:824] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:39.035024 28925 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:39.035285 28925 ts_tablet_manager.cc:1367] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.012s
I20240503 14:15:39.035486 28923 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:39.035843 28923 ts_tablet_manager.cc:1367] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.008s
I20240503 14:15:39.036125 28935 raft_consensus.cc:729] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.036437 28925 raft_consensus.cc:348] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.036841 28925 raft_consensus.cc:374] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.036964 28925 raft_consensus.cc:729] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.039511 28935 consensus_queue.cc:260] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.042754 28934 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:39.043136 28934 ts_tablet_manager.cc:1367] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.003s
I20240503 14:15:39.047338 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:39.047957 28188 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:39.050395 28925 consensus_queue.cc:260] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.051142 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:39.052575 28925 ts_tablet_manager.cc:1398] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240503 14:15:39.052799 28943 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:39.056946 28943 tablet_bootstrap.cc:654] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.057680 28923 raft_consensus.cc:348] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.058228 28923 raft_consensus.cc:374] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:15:39.052742 27860 leader_election.cc:341] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.061115 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:39.058332 28923 raft_consensus.cc:729] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.061746 28057 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:39.063097 28929 raft_consensus.cc:482] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:39.063314 28929 raft_consensus.cc:504] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.063417 28923 consensus_queue.cc:260] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.064478 28929 leader_election.cc:288] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:15:39.064208 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:39.065295 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
W20240503 14:15:39.065945 27860 leader_election.cc:341] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.065613 28934 raft_consensus.cc:348] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.066008 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:39.066159 28934 raft_consensus.cc:374] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.066234 28057 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:39.066277 28934 raft_consensus.cc:729] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.067066 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:39.067220 28934 consensus_queue.cc:260] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.069159 28935 ts_tablet_manager.cc:1398] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.072s user 0.015s sys 0.000s
I20240503 14:15:39.069732 28935 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:39.071106 28934 ts_tablet_manager.cc:1398] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.028s user 0.010s sys 0.004s
I20240503 14:15:39.069448 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:39.071556 28934 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:39.072017 28935 tablet_bootstrap.cc:654] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.067653 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:39.075482 28943 log.cc:824] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:39.075663 28923 ts_tablet_manager.cc:1398] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.040s user 0.012s sys 0.004s
I20240503 14:15:39.076082 28923 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:39.077720 28923 tablet_bootstrap.cc:654] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.077952 28937 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:39.078282 28937 ts_tablet_manager.cc:1367] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.073s user 0.019s sys 0.011s
I20240503 14:15:39.078675 28934 tablet_bootstrap.cc:654] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
W20240503 14:15:39.081914 27861 leader_election.cc:341] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:15:39.082221 27861 leader_election.cc:341] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.084966 28934 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:39.085147 28934 ts_tablet_manager.cc:1367] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240503 14:15:39.084923 28937 raft_consensus.cc:348] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.085451 28937 raft_consensus.cc:374] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.085552 28937 raft_consensus.cc:729] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.086386 28937 consensus_queue.cc:260] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.087805 28934 raft_consensus.cc:348] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.088109 28934 raft_consensus.cc:374] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.088224 28934 raft_consensus.cc:729] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
W20240503 14:15:39.089682 27862 leader_election.cc:341] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:15:39.089947 27862 leader_election.cc:341] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.090008 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:39.090528 28923 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:39.090718 28923 ts_tablet_manager.cc:1367] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
W20240503 14:15:39.091897 28103 tablet.cc:2369] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:39.092855 28935 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:39.093043 28935 ts_tablet_manager.cc:1367] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.003s
I20240503 14:15:39.094192 28935 raft_consensus.cc:348] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.094486 28935 raft_consensus.cc:374] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.094594 28935 raft_consensus.cc:729] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.094878 28935 consensus_queue.cc:260] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.091825 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:39.095988 28935 ts_tablet_manager.cc:1398] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:39.091778 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:39.097937 28935 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:39.097949 28948 raft_consensus.cc:482] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:39.098234 28948 raft_consensus.cc:504] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.100172 28943 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:39.100697 28943 ts_tablet_manager.cc:1367] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.006s
I20240503 14:15:39.104493 28935 tablet_bootstrap.cc:654] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.106608 28948 leader_election.cc:288] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:39.112949 28943 raft_consensus.cc:348] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
W20240503 14:15:39.115962 28497 tablet.cc:2369] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:39.116209 28943 raft_consensus.cc:374] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.116541 28943 raft_consensus.cc:729] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.118063 28928 raft_consensus.cc:482] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:39.118292 28928 raft_consensus.cc:504] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
W20240503 14:15:39.121794 28234 tablet.cc:2369] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:15:39.122366 27840 tablet.cc:2369] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:39.129468 28928 leader_election.cc:288] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
W20240503 14:15:39.133699 27862 leader_election.cc:341] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:15:39.133728 27972 tablet.cc:2369] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:39.133940 27862 leader_election.cc:302] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, a242f99f54b4494893064cfe3030b4ad; no voters: 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422
I20240503 14:15:39.134205 28935 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
W20240503 14:15:39.134363 27861 leader_election.cc:341] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.134418 28935 ts_tablet_manager.cc:1367] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.004s
I20240503 14:15:39.133772 28929 raft_consensus.cc:482] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:39.134887 28929 raft_consensus.cc:504] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
W20240503 14:15:39.135385 27861 leader_election.cc:341] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.135236 28935 raft_consensus.cc:348] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.135574 28935 raft_consensus.cc:374] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.135529 27861 leader_election.cc:302] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:39.135661 28935 raft_consensus.cc:729] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.136161 28929 leader_election.cc:288] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:15:39.136211 28935 consensus_queue.cc:260] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.136360 28929 raft_consensus.cc:2726] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:39.138083 28935 ts_tablet_manager.cc:1398] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:15:39.138849 28943 consensus_queue.cc:260] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.140645 28929 raft_consensus.cc:2726] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:15:39.141887 27862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.145.2:38859, user_credentials={real_user=slave}} blocked reactor thread for 50677.6us
I20240503 14:15:39.144042 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:39.144143 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:39.144312 28188 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:39.144498 27794 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:39.129071 28923 raft_consensus.cc:348] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.147277 28923 raft_consensus.cc:374] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.147408 28923 raft_consensus.cc:729] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.147691 28923 consensus_queue.cc:260] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.148535 28923 ts_tablet_manager.cc:1398] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.058s user 0.002s sys 0.001s
I20240503 14:15:39.149195 28923 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:39.152797 28923 tablet_bootstrap.cc:654] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.168711 28934 consensus_queue.cc:260] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.169718 28934 ts_tablet_manager.cc:1398] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.084s user 0.001s sys 0.002s
I20240503 14:15:39.170071 28934 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:39.172431 28934 tablet_bootstrap.cc:654] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
W20240503 14:15:39.181663 27861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.145.1:35261, user_credentials={real_user=slave}} blocked reactor thread for 46277.6us
I20240503 14:15:39.183483 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:39.183636 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:39.188563 28947 raft_consensus.cc:482] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:39.188843 28947 raft_consensus.cc:504] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.191131 28937 ts_tablet_manager.cc:1398] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.113s user 0.017s sys 0.005s
I20240503 14:15:39.191875 28937 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
W20240503 14:15:39.195411 27861 leader_election.cc:341] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:15:39.196496 27861 leader_election.cc:341] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.202805 28923 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:39.202936 28947 leader_election.cc:288] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:39.203043 28923 ts_tablet_manager.cc:1367] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20240503 14:15:39.203902 28923 raft_consensus.cc:348] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.204269 28923 raft_consensus.cc:374] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.204363 28923 raft_consensus.cc:729] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.205114 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:39.205394 28451 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:39.210217 28923 consensus_queue.cc:260] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.218092 28923 ts_tablet_manager.cc:1398] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240503 14:15:39.223536 28943 ts_tablet_manager.cc:1398] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.123s user 0.036s sys 0.014s
I20240503 14:15:39.224169 27860 leader_election.cc:302] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 7046cbd82d634976b53fe0cef5788f18, acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:39.224997 28962 raft_consensus.cc:2781] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:39.225154 28962 raft_consensus.cc:482] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:39.225131 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:39.225360 28962 raft_consensus.cc:3037] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:15:39.225981 28386 leader_election.cc:341] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.228775 28962 raft_consensus.cc:504] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.230578 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
W20240503 14:15:39.231271 27861 leader_election.cc:341] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.232334 28943 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:39.235286 28943 tablet_bootstrap.cc:654] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.240206 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:39.240557 28451 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:39.241986 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:39.242209 28451 raft_consensus.cc:3037] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.242784 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:39.243217 28962 leader_election.cc:288] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 election: Requested vote from peers 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
W20240503 14:15:39.243422 27861 leader_election.cc:341] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.245645 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:39.245956 28188 raft_consensus.cc:3037] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.246024 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:39.246261 27794 raft_consensus.cc:3037] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.249246 27794 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 1.
I20240503 14:15:39.249233 28188 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 1.
I20240503 14:15:39.249519 28451 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 1.
I20240503 14:15:39.250133 27862 leader_election.cc:302] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 7046cbd82d634976b53fe0cef5788f18, acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:39.250524 28962 raft_consensus.cc:2781] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:39.250735 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:39.250838 28962 raft_consensus.cc:686] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Becoming Leader. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Running, Role: LEADER
W20240503 14:15:39.251382 28386 leader_election.cc:341] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.251509 28962 consensus_queue.cc:237] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.251838 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:39.252084 28057 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:39.255897 28934 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:39.256136 28934 ts_tablet_manager.cc:1367] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.003s
I20240503 14:15:39.257009 28934 raft_consensus.cc:348] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.257448 28934 raft_consensus.cc:374] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.257534 28934 raft_consensus.cc:729] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.257838 28934 consensus_queue.cc:260] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.261804 28934 ts_tablet_manager.cc:1398] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:15:39.262828 28943 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:39.265014 28943 ts_tablet_manager.cc:1367] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.003s
I20240503 14:15:39.266856 28937 tablet_bootstrap.cc:654] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.267347 28954 raft_consensus.cc:482] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:39.267642 28954 raft_consensus.cc:504] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.269992 28943 raft_consensus.cc:348] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.272755 28943 raft_consensus.cc:374] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.273506 28943 raft_consensus.cc:729] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.274763 28943 consensus_queue.cc:260] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.278057 28943 ts_tablet_manager.cc:1398] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240503 14:15:39.280007 28943 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:39.281572 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:39.283224 28954 leader_election.cc:288] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:39.297326 28943 tablet_bootstrap.cc:654] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.298601 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:39.299043 27794 raft_consensus.cc:2379] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 078f8eff84c94dde9705ad1330922a11 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:39.301156 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:39.301582 28188 raft_consensus.cc:2379] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9231e715a4e342a197d6c5456ba6b399 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:39.302572 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:39.302886 28057 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:39.302734 28056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:39.303081 28056 raft_consensus.cc:3037] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.306262 28056 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 1.
I20240503 14:15:39.308694 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:39.308992 28188 raft_consensus.cc:2379] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 078f8eff84c94dde9705ad1330922a11 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:39.308938 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
W20240503 14:15:39.309829 27992 leader_election.cc:341] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.310017 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
W20240503 14:15:39.310539 27992 leader_election.cc:341] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:39.310693 27992 leader_election.cc:302] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 9231e715a4e342a197d6c5456ba6b399; no voters: 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:39.311223 28928 raft_consensus.cc:2726] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:39.313694 28943 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:39.313891 28943 ts_tablet_manager.cc:1367] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.003s
I20240503 14:15:39.314750 28943 raft_consensus.cc:348] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.315064 28943 raft_consensus.cc:374] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.315155 28943 raft_consensus.cc:729] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.315434 28943 consensus_queue.cc:260] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.318333 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:39.318610 27794 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:39.318758 28943 ts_tablet_manager.cc:1398] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:15:39.326752 28937 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:39.326952 28937 ts_tablet_manager.cc:1367] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.135s user 0.004s sys 0.004s
I20240503 14:15:39.328190 28937 raft_consensus.cc:348] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.328555 28937 raft_consensus.cc:374] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.328667 28937 raft_consensus.cc:729] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.328925 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:39.328922 28937 consensus_queue.cc:260] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.329169 28451 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:39.328846 27520 catalog_manager.cc:5484] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad reported cstate change: term changed from 0 to 1, leader changed from <none> to a242f99f54b4494893064cfe3030b4ad (127.10.145.3). New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:39.329721 28937 ts_tablet_manager.cc:1398] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:15:39.329887 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:39.330106 28937 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:39.330168 27794 raft_consensus.cc:2379] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9231e715a4e342a197d6c5456ba6b399 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:39.330828 28387 leader_election.cc:302] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 9231e715a4e342a197d6c5456ba6b399; no voters: 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422
I20240503 14:15:39.331297 28948 raft_consensus.cc:2726] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:39.332290 28937 tablet_bootstrap.cc:654] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.332391 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:39.334319 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:39.334559 28451 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:39.340070 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:39.340541 27662 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:39.341272 28123 leader_election.cc:302] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:39.341701 28937 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:39.341907 28937 ts_tablet_manager.cc:1367] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20240503 14:15:39.342005 28947 raft_consensus.cc:2781] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:39.342134 28947 raft_consensus.cc:482] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:39.342219 28947 raft_consensus.cc:3037] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.342813 28937 raft_consensus.cc:348] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.343106 28937 raft_consensus.cc:374] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.343189 28937 raft_consensus.cc:729] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.343494 28937 consensus_queue.cc:260] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.344262 28937 ts_tablet_manager.cc:1398] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:39.346004 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:39.346297 27794 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:39.346863 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:39.347128 28320 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:39.347455 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:39.347702 28188 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:39.345185 28947 raft_consensus.cc:504] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.349794 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:39.350075 28451 raft_consensus.cc:3037] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.350901 28256 leader_election.cc:302] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:39.351712 28954 raft_consensus.cc:2781] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:39.351873 28954 raft_consensus.cc:482] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:39.351975 28954 raft_consensus.cc:3037] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.351931 28056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:39.352180 28056 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:39.353754 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:39.354027 27662 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:39.353212 28451 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:39.355135 28954 raft_consensus.cc:504] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
W20240503 14:15:39.355831 27708 tablet.cc:2369] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:39.356288 28056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:39.356423 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:39.356551 28056 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:39.356796 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:39.357004 28451 raft_consensus.cc:3037] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.357980 28947 leader_election.cc:288] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 election: Requested vote from peers 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:39.358271 28954 leader_election.cc:288] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 election: Requested vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:39.358824 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:39.358906 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:39.359113 28057 raft_consensus.cc:3037] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.359078 27661 raft_consensus.cc:3037] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.360357 28451 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
W20240503 14:15:39.361449 28366 tablet.cc:2369] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:39.361794 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:39.361959 28057 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:39.361928 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:39.362038 28320 raft_consensus.cc:3037] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.362181 27794 raft_consensus.cc:3037] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.362874 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:39.363145 27926 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:39.363261 27925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:39.363534 27925 raft_consensus.cc:3037] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.364936 28320 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:39.365654 27794 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:39.366243 28256 leader_election.cc:302] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, acb93936f2f14e628eecb05d7d1f4d91; no voters:
I20240503 14:15:39.366498 27925 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:39.366587 27661 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:39.366762 28954 raft_consensus.cc:2781] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:39.366930 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:39.367209 27662 raft_consensus.cc:3037] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.367787 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:39.368095 27794 raft_consensus.cc:3037] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.367839 28123 leader_election.cc:302] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 078f8eff84c94dde9705ad1330922a11
I20240503 14:15:39.368489 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add941da26824cdda7489b5da990e879" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:39.368618 28947 raft_consensus.cc:2781] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:39.368718 28188 raft_consensus.cc:3037] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.369946 27662 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:39.371798 28188 raft_consensus.cc:2454] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:39.372695 27794 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:39.374498 28954 raft_consensus.cc:686] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Becoming Leader. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Running, Role: LEADER
I20240503 14:15:39.375224 28954 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.376333 28947 raft_consensus.cc:686] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Becoming Leader. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Running, Role: LEADER
I20240503 14:15:39.376976 28947 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.377611 27925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:39.377878 27925 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:39.377658 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dae8a392374b91be378ec431838686" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:39.378156 27926 raft_consensus.cc:3037] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.381126 27926 raft_consensus.cc:2454] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:39.384443 27519 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: term changed from 0 to 1, leader changed from <none> to c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5). New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:39.386150 27520 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6). New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:39.455727 28963 raft_consensus.cc:482] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:39.455955 28963 raft_consensus.cc:504] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:39.458887 28963 leader_election.cc:288] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:39.469862 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:39.470149 28320 raft_consensus.cc:2454] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:39.474658 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:39.475016 28188 raft_consensus.cc:2379] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7046cbd82d634976b53fe0cef5788f18 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:39.475394 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:39.475519 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:39.475687 27794 raft_consensus.cc:2379] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7046cbd82d634976b53fe0cef5788f18 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:39.475850 28451 raft_consensus.cc:2379] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7046cbd82d634976b53fe0cef5788f18 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:39.476078 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:39.476369 28057 raft_consensus.cc:2379] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7046cbd82d634976b53fe0cef5788f18 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:39.476778 27598 leader_election.cc:302] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 7046cbd82d634976b53fe0cef5788f18, acb93936f2f14e628eecb05d7d1f4d91; no voters: 078f8eff84c94dde9705ad1330922a11, 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, c91f285b8f4f4a27bcdd0ada303c1422
I20240503 14:15:39.477165 28963 raft_consensus.cc:3037] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.479409 28963 raft_consensus.cc:2726] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:39.480814 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae969579ee545c9838d3c26e4abd66f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:39.640919 28962 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.657655 28320 raft_consensus.cc:3037] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.660995 28962 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.729697 28962 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.730844 28962 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.749359 28962 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.751607 29010 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.792258 28947 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.822917 28954 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:39.850443 29005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:39.850895 29005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:39.859035 29005 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:39.864322 28954 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.877117 28954 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.880532 28954 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.898361 29039 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.898764 28947 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.899688 29034 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 14:15:39.904278 28947 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.936231 28947 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.947732 28057 raft_consensus.cc:3037] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.953832 28947 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.983695 29043 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 413.482 ms
time per row: 0.206741 ms
I20240503 14:15:40.548853 27519 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51414:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:40.549656 27519 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 14:15:40.559315 28167 tablet_service.cc:1467] Processing CreateTablet for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.559510 28036 tablet_service.cc:1467] Processing CreateTablet for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.559885 28167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fb4ed8413f4eceb4348f9d216d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.559958 28036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fb4ed8413f4eceb4348f9d216d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.560015 28429 tablet_service.cc:1467] Processing CreateTablet for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.560472 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fb4ed8413f4eceb4348f9d216d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.564355 28299 tablet_service.cc:1467] Processing CreateTablet for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.564811 28299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fb4ed8413f4eceb4348f9d216d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.566334 27641 tablet_service.cc:1467] Processing CreateTablet for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.566828 27641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fb4ed8413f4eceb4348f9d216d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.569411 28037 tablet_service.cc:1467] Processing CreateTablet for tablet 6b178a32f4b24249aeae16cada4a3d56 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.569896 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b178a32f4b24249aeae16cada4a3d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.570219 27772 tablet_service.cc:1467] Processing CreateTablet for tablet 6b178a32f4b24249aeae16cada4a3d56 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.570695 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b178a32f4b24249aeae16cada4a3d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.570806 27905 tablet_service.cc:1467] Processing CreateTablet for tablet 6b178a32f4b24249aeae16cada4a3d56 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.571286 27905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b178a32f4b24249aeae16cada4a3d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.573774 27773 tablet_service.cc:1467] Processing CreateTablet for tablet f15a6ff6acc242258172e585b1ea34c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.574224 27773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15a6ff6acc242258172e585b1ea34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.574218 28298 tablet_service.cc:1467] Processing CreateTablet for tablet 6b178a32f4b24249aeae16cada4a3d56 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.574362 27906 tablet_service.cc:1467] Processing CreateTablet for tablet f15a6ff6acc242258172e585b1ea34c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.574645 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b178a32f4b24249aeae16cada4a3d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.574798 27906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15a6ff6acc242258172e585b1ea34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.576726 28166 tablet_service.cc:1467] Processing CreateTablet for tablet 6b178a32f4b24249aeae16cada4a3d56 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.577224 28166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b178a32f4b24249aeae16cada4a3d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.577672 28300 tablet_service.cc:1467] Processing CreateTablet for tablet f15a6ff6acc242258172e585b1ea34c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.578061 28300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15a6ff6acc242258172e585b1ea34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.578686 28431 tablet_service.cc:1467] Processing CreateTablet for tablet f15a6ff6acc242258172e585b1ea34c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.579111 28431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15a6ff6acc242258172e585b1ea34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.581725 27904 tablet_service.cc:1467] Processing CreateTablet for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.581750 28168 tablet_service.cc:1467] Processing CreateTablet for tablet f15a6ff6acc242258172e585b1ea34c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.582149 27904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fb4ed8413f4eceb4348f9d216d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.582233 28168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15a6ff6acc242258172e585b1ea34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.583333 28430 tablet_service.cc:1467] Processing CreateTablet for tablet 6b178a32f4b24249aeae16cada4a3d56 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.583747 28430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b178a32f4b24249aeae16cada4a3d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.584798 28035 tablet_service.cc:1467] Processing CreateTablet for tablet f15a6ff6acc242258172e585b1ea34c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.566851 27642 tablet_service.cc:1467] Processing CreateTablet for tablet f15a6ff6acc242258172e585b1ea34c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.585592 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15a6ff6acc242258172e585b1ea34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.585647 27642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15a6ff6acc242258172e585b1ea34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.589547 27774 tablet_service.cc:1467] Processing CreateTablet for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.590127 27774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fb4ed8413f4eceb4348f9d216d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.566295 27640 tablet_service.cc:1467] Processing CreateTablet for tablet 6b178a32f4b24249aeae16cada4a3d56 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.591187 27640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b178a32f4b24249aeae16cada4a3d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.610105 29106 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:40.612059 29106 tablet_bootstrap.cc:654] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.616703 29109 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:40.618700 29109 tablet_bootstrap.cc:654] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.623678 29106 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:40.623996 29106 ts_tablet_manager.cc:1367] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240503 14:15:40.625015 29106 raft_consensus.cc:348] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.625563 29106 raft_consensus.cc:374] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.625741 29106 raft_consensus.cc:729] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.626158 29106 consensus_queue.cc:260] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.628329 29106 ts_tablet_manager.cc:1398] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:15:40.628777 29106 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:40.634475 29106 tablet_bootstrap.cc:654] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.635488 29111 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:40.639194 29111 tablet_bootstrap.cc:654] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.642999 29111 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:40.648177 29111 ts_tablet_manager.cc:1367] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 14:15:40.651038 29111 raft_consensus.cc:348] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.651512 29104 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:40.651522 29106 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:40.651736 29106 ts_tablet_manager.cc:1367] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.008s
I20240503 14:15:40.652145 29111 raft_consensus.cc:374] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.652279 29111 raft_consensus.cc:729] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.652580 29111 consensus_queue.cc:260] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.652578 29106 raft_consensus.cc:348] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.652904 29106 raft_consensus.cc:374] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.652976 29106 raft_consensus.cc:729] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.655146 29104 tablet_bootstrap.cc:654] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.655180 29106 consensus_queue.cc:260] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.659430 29106 ts_tablet_manager.cc:1398] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20240503 14:15:40.662895 29106 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:40.664611 29106 tablet_bootstrap.cc:654] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.667598 29111 ts_tablet_manager.cc:1398] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20240503 14:15:40.667946 29111 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:40.670264 29111 tablet_bootstrap.cc:654] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.672994 29104 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:40.674507 29104 ts_tablet_manager.cc:1367] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.007s
I20240503 14:15:40.675195 29116 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:40.676507 29104 raft_consensus.cc:348] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.679322 29116 tablet_bootstrap.cc:654] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.679912 29104 raft_consensus.cc:374] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.679719 29107 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:40.680105 29104 raft_consensus.cc:729] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.680434 29104 consensus_queue.cc:260] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.681378 29104 ts_tablet_manager.cc:1398] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:15:40.681905 29106 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:40.682083 29106 ts_tablet_manager.cc:1367] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20240503 14:15:40.682363 29104 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:40.682888 29106 raft_consensus.cc:348] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.683159 29106 raft_consensus.cc:374] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.683256 29106 raft_consensus.cc:729] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.684831 29104 tablet_bootstrap.cc:654] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.685547 29106 consensus_queue.cc:260] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.687575 29107 tablet_bootstrap.cc:654] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.687924 29116 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:40.688097 29116 ts_tablet_manager.cc:1367] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 14:15:40.688946 29116 raft_consensus.cc:348] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.689175 29106 ts_tablet_manager.cc:1398] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240503 14:15:40.689260 29116 raft_consensus.cc:374] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.689381 29116 raft_consensus.cc:729] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.690404 29109 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:40.690652 29109 ts_tablet_manager.cc:1367] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.074s user 0.014s sys 0.009s
I20240503 14:15:40.691668 29109 raft_consensus.cc:348] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.692157 29109 raft_consensus.cc:374] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.692296 29109 raft_consensus.cc:729] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.692636 29111 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:40.692847 29111 ts_tablet_manager.cc:1367] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.016s
I20240503 14:15:40.692656 29109 consensus_queue.cc:260] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.693696 29109 ts_tablet_manager.cc:1398] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:40.693773 29111 raft_consensus.cc:348] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.694010 29105 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:40.694072 29111 raft_consensus.cc:374] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.694149 29111 raft_consensus.cc:729] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.694269 29109 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:40.694414 29111 consensus_queue.cc:260] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.695727 29105 tablet_bootstrap.cc:654] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.695899 29111 ts_tablet_manager.cc:1398] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:40.696193 29111 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:40.697268 29104 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:40.697733 29107 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:40.697926 29107 ts_tablet_manager.cc:1367] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 14:15:40.698050 29104 ts_tablet_manager.cc:1367] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 14:15:40.698297 29111 tablet_bootstrap.cc:654] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.697567 29116 consensus_queue.cc:260] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.699066 29104 raft_consensus.cc:348] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.700009 29104 raft_consensus.cc:374] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.700147 29104 raft_consensus.cc:729] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.700512 29104 consensus_queue.cc:260] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.700835 29109 tablet_bootstrap.cc:654] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.701417 29104 ts_tablet_manager.cc:1398] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:40.701812 29104 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:40.702069 29107 raft_consensus.cc:348] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.702381 29107 raft_consensus.cc:374] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.702497 29107 raft_consensus.cc:729] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.702782 29107 consensus_queue.cc:260] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.704123 29104 tablet_bootstrap.cc:654] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.705360 29105 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:40.705590 29105 ts_tablet_manager.cc:1367] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 14:15:40.706559 29105 raft_consensus.cc:348] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.706988 29105 raft_consensus.cc:374] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.707118 29105 raft_consensus.cc:729] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.707540 29105 consensus_queue.cc:260] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.709386 29116 ts_tablet_manager.cc:1398] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20240503 14:15:40.710240 29116 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:40.714325 29116 tablet_bootstrap.cc:654] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.716105 29105 ts_tablet_manager.cc:1398] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:15:40.714687 29048 raft_consensus.cc:482] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:40.716436 29048 raft_consensus.cc:504] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.717339 29105 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:40.717882 29048 leader_election.cc:288] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:40.720309 29105 tablet_bootstrap.cc:654] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.720963 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:40.721596 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:40.721845 27926 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:40.722311 29116 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:40.722281 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:40.722510 29116 ts_tablet_manager.cc:1367] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 14:15:40.722833 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:40.723371 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:40.723364 29116 raft_consensus.cc:348] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.723654 29116 raft_consensus.cc:374] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.723737 29116 raft_consensus.cc:729] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.723994 29116 consensus_queue.cc:260] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
W20240503 14:15:40.724187 28123 leader_election.cc:341] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:40.724629 29116 ts_tablet_manager.cc:1398] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240503 14:15:40.724954 28122 leader_election.cc:341] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:40.725023 29116 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
W20240503 14:15:40.725621 28124 leader_election.cc:341] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:40.720615 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:40.725970 28057 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:40.726831 29107 ts_tablet_manager.cc:1398] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
W20240503 14:15:40.726887 28123 leader_election.cc:341] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:40.727017 28123 leader_election.cc:302] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:40.727236 29107 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:40.727896 29048 raft_consensus.cc:2726] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:40.728228 29124 raft_consensus.cc:482] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:40.728430 29124 raft_consensus.cc:504] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.728983 29107 tablet_bootstrap.cc:654] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.729861 29104 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:40.729939 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:40.730120 29104 ts_tablet_manager.cc:1367] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20240503 14:15:40.730187 27793 raft_consensus.cc:2454] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:40.730715 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:40.730969 27926 raft_consensus.cc:2454] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:40.731199 29104 raft_consensus.cc:348] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.731470 29104 raft_consensus.cc:374] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.731570 29104 raft_consensus.cc:729] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.731824 29104 consensus_queue.cc:260] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.732103 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:40.732336 28187 raft_consensus.cc:2454] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:40.732589 29116 tablet_bootstrap.cc:654] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.732615 29104 ts_tablet_manager.cc:1398] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:40.732823 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:40.734026 29111 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:40.734241 29111 ts_tablet_manager.cc:1367] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.003s
I20240503 14:15:40.734426 29105 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:40.734613 29105 ts_tablet_manager.cc:1367] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 14:15:40.735242 29111 raft_consensus.cc:348] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.735360 29105 raft_consensus.cc:348] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.735632 29105 raft_consensus.cc:374] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.735572 29111 raft_consensus.cc:374] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.735715 29105 raft_consensus.cc:729] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.735810 29111 raft_consensus.cc:729] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.736003 29105 consensus_queue.cc:260] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.736121 29111 consensus_queue.cc:260] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
W20240503 14:15:40.736415 27597 leader_election.cc:341] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:40.736704 29105 ts_tablet_manager.cc:1398] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:40.736874 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:40.737027 29105 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:40.737473 29111 ts_tablet_manager.cc:1398] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:40.738672 29105 tablet_bootstrap.cc:654] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
W20240503 14:15:40.738744 27598 leader_election.cc:341] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:40.738974 27598 leader_election.cc:302] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 078f8eff84c94dde9705ad1330922a11, acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:40.739863 29133 raft_consensus.cc:2781] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:40.740803 29133 raft_consensus.cc:482] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:40.741014 29133 raft_consensus.cc:3037] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.744439 29133 raft_consensus.cc:504] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.745107 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
W20240503 14:15:40.745803 27596 leader_election.cc:341] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:40.746739 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:40.746913 29105 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:40.746955 28187 raft_consensus.cc:3037] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.747066 29105 ts_tablet_manager.cc:1367] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240503 14:15:40.747258 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:40.747707 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:40.747913 27926 raft_consensus.cc:3037] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:15:40.747951 27598 leader_election.cc:341] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:40.748492 29105 raft_consensus.cc:348] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.748741 29105 raft_consensus.cc:374] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.748824 29105 raft_consensus.cc:729] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.749140 29105 consensus_queue.cc:260] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.749673 28187 raft_consensus.cc:2454] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:40.750133 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:40.750355 27793 raft_consensus.cc:3037] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.750514 29105 ts_tablet_manager.cc:1398] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:40.752496 27793 raft_consensus.cc:2454] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:40.752549 27926 raft_consensus.cc:2454] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:40.753002 29124 leader_election.cc:288] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:40.753475 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
W20240503 14:15:40.754446 27597 leader_election.cc:341] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:40.754801 29109 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:40.754822 27598 leader_election.cc:302] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 078f8eff84c94dde9705ad1330922a11, acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:40.755127 29109 ts_tablet_manager.cc:1367] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.005s
I20240503 14:15:40.755151 29124 raft_consensus.cc:2781] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:40.755931 29109 raft_consensus.cc:348] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.756239 29109 raft_consensus.cc:374] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.756327 29109 raft_consensus.cc:729] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.756726 29107 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:40.756815 29116 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:40.756654 29109 consensus_queue.cc:260] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.756879 29124 raft_consensus.cc:686] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Becoming Leader. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Running, Role: LEADER
I20240503 14:15:40.756984 29116 ts_tablet_manager.cc:1367] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.000s
I20240503 14:15:40.757126 29107 ts_tablet_manager.cc:1367] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.010s
I20240503 14:15:40.757531 29124 consensus_queue.cc:237] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.757923 29116 raft_consensus.cc:348] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.758198 29116 raft_consensus.cc:374] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.758313 29116 raft_consensus.cc:729] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.758066 29107 raft_consensus.cc:348] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.758627 29116 consensus_queue.cc:260] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.758796 29107 raft_consensus.cc:374] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.758968 29107 raft_consensus.cc:729] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.759279 29116 ts_tablet_manager.cc:1398] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:40.759272 29107 consensus_queue.cc:260] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.759862 29109 ts_tablet_manager.cc:1398] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:15:40.760213 29109 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:40.760757 29107 ts_tablet_manager.cc:1398] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:40.761144 29107 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:40.761945 29133 leader_election.cc:288] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Requested vote from peers 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:40.762436 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
W20240503 14:15:40.763052 27596 leader_election.cc:341] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:40.763778 29109 tablet_bootstrap.cc:654] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.763823 29107 tablet_bootstrap.cc:654] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.769913 29107 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:40.769632 27514 catalog_manager.cc:5484] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:40.770097 29109 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:40.770102 29107 ts_tablet_manager.cc:1367] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240503 14:15:40.770252 29109 ts_tablet_manager.cc:1367] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20240503 14:15:40.770916 29107 raft_consensus.cc:348] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.770978 29109 raft_consensus.cc:348] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.771214 29109 raft_consensus.cc:374] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.771250 29107 raft_consensus.cc:374] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.771322 29109 raft_consensus.cc:729] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.771418 29107 raft_consensus.cc:729] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.771642 29109 consensus_queue.cc:260] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.771672 29107 consensus_queue.cc:260] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.772269 29109 ts_tablet_manager.cc:1398] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:40.772346 29107 ts_tablet_manager.cc:1398] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:40.787299 29127 raft_consensus.cc:482] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:40.787524 29127 raft_consensus.cc:504] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.788910 29127 leader_election.cc:288] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:40.788929 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:40.789188 28188 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:40.789060 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:40.789330 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:40.789402 28320 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:40.789572 27661 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:40.789937 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:40.790100 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:40.790170 27793 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:40.790328 28451 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:40.790292 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:40.790526 27926 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:40.790580 27992 leader_election.cc:302] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:40.790891 29127 raft_consensus.cc:2781] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:40.791023 29127 raft_consensus.cc:482] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:40.791136 29127 raft_consensus.cc:3037] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.793133 29127 raft_consensus.cc:504] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.794380 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:40.794382 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:40.794481 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:40.794565 28320 raft_consensus.cc:3037] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.794593 28187 raft_consensus.cc:3037] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.794670 28451 raft_consensus.cc:3037] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.795029 29127 leader_election.cc:288] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 election: Requested vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:40.796763 28320 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:40.797727 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:40.797784 28451 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:40.797976 27661 raft_consensus.cc:3037] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.799055 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:40.799305 27926 raft_consensus.cc:3037] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.799252 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b178a32f4b24249aeae16cada4a3d56" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:40.799458 27793 raft_consensus.cc:3037] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.800746 27661 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:40.801304 27926 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:40.801343 27992 leader_election.cc:302] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, acb93936f2f14e628eecb05d7d1f4d91; no voters:
I20240503 14:15:40.801461 28187 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:40.801671 29127 raft_consensus.cc:2781] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:40.802189 27793 raft_consensus.cc:2454] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:40.802660 29127 raft_consensus.cc:686] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Becoming Leader. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Running, Role: LEADER
I20240503 14:15:40.803185 29127 consensus_queue.cc:237] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.807150 27519 catalog_manager.cc:5484] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 078f8eff84c94dde9705ad1330922a11 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:40.902647 28954 raft_consensus.cc:482] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:40.902842 28954 raft_consensus.cc:504] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:40.903918 28954 leader_election.cc:288] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:40.904233 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:40.904531 27794 raft_consensus.cc:2379] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acb93936f2f14e628eecb05d7d1f4d91 in current term 1: Already voted for candidate 7046cbd82d634976b53fe0cef5788f18 in this term.
I20240503 14:15:40.904533 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:40.904523 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:40.904573 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:40.904774 28187 raft_consensus.cc:2379] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acb93936f2f14e628eecb05d7d1f4d91 in current term 1: Already voted for candidate 7046cbd82d634976b53fe0cef5788f18 in this term.
I20240503 14:15:40.904878 27926 raft_consensus.cc:2379] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acb93936f2f14e628eecb05d7d1f4d91 in current term 1: Already voted for candidate 7046cbd82d634976b53fe0cef5788f18 in this term.
I20240503 14:15:40.905401 28256 leader_election.cc:302] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: acb93936f2f14e628eecb05d7d1f4d91; no voters: 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422
I20240503 14:15:40.905447 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:40.905699 28451 raft_consensus.cc:2454] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:40.905696 28954 raft_consensus.cc:3037] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.905839 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15a6ff6acc242258172e585b1ea34c6" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:40.906069 28057 raft_consensus.cc:2454] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:40.907914 28954 raft_consensus.cc:2726] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:40.977373 29125 raft_consensus.cc:482] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:40.977578 29125 raft_consensus.cc:504] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.978701 29125 leader_election.cc:288] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:40.979218 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:40.979277 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:40.979414 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:40.979506 28057 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:40.979521 27926 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:40.979480 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:40.979635 28187 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:40.979734 28320 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:40.980057 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:40.980116 28387 leader_election.cc:302] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 9231e715a4e342a197d6c5456ba6b399, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:40.980100 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:40.980307 27794 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:40.980319 27661 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:40.980463 29125 raft_consensus.cc:2781] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:40.980592 29125 raft_consensus.cc:482] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:40.980718 29125 raft_consensus.cc:3037] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.983021 29125 raft_consensus.cc:504] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.984297 29125 leader_election.cc:288] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 election: Requested vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:40.984282 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:40.984407 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:40.984637 28057 raft_consensus.cc:3037] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.984637 28187 raft_consensus.cc:3037] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.984637 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:40.984876 27661 raft_consensus.cc:3037] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.985042 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:40.985270 28320 raft_consensus.cc:3037] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.985311 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:40.985513 27926 raft_consensus.cc:3037] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.987429 28057 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:40.987716 27661 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:40.987847 28320 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:40.988121 27926 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:40.988199 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fb4ed8413f4eceb4348f9d216d3dd7" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:40.988325 28386 leader_election.cc:302] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, acb93936f2f14e628eecb05d7d1f4d91; no voters:
I20240503 14:15:40.988420 27794 raft_consensus.cc:3037] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.988673 29125 raft_consensus.cc:2781] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:40.989106 29125 raft_consensus.cc:686] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Becoming Leader. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Running, Role: LEADER
I20240503 14:15:40.989737 29125 consensus_queue.cc:237] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:40.990525 27794 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:40.991523 28187 raft_consensus.cc:2454] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:40.994586 27519 catalog_manager.cc:5484] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7). New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:41.208503 29124 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.235443 29124 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.241770 29127 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.302865 28057 raft_consensus.cc:3037] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:41.306885 29124 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.312781 28451 raft_consensus.cc:3037] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:41.313967 29151 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.316732 29124 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.323588 29124 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.380730 29127 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.406778 29127 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.409956 29127 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.416886 29125 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.436559 29169 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.447448 29168 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.481362 29125 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:41.492770 29143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:41.493158 29143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:41.500825 29143 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 14:15:41.546676 29125 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.555296 29192 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:15:41.558126 29125 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.561959 29197 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 376.956 ms
time per row: 0.188478 ms
I20240503 14:15:42.054998 27514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51414:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:42.055740 27514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 14:15:42.065832 27774 tablet_service.cc:1467] Processing CreateTablet for tablet 233ee592f6e843c8a68a5170e0f9d66d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.065935 27642 tablet_service.cc:1467] Processing CreateTablet for tablet 233ee592f6e843c8a68a5170e0f9d66d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.066352 27774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ee592f6e843c8a68a5170e0f9d66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.066352 27642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ee592f6e843c8a68a5170e0f9d66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.066887 28036 tablet_service.cc:1467] Processing CreateTablet for tablet 233ee592f6e843c8a68a5170e0f9d66d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.067366 28036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ee592f6e843c8a68a5170e0f9d66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.072676 29238 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:42.072692 27906 tablet_service.cc:1467] Processing CreateTablet for tablet 233ee592f6e843c8a68a5170e0f9d66d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.073347 27906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ee592f6e843c8a68a5170e0f9d66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.073530 28167 tablet_service.cc:1467] Processing CreateTablet for tablet 233ee592f6e843c8a68a5170e0f9d66d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.074030 28167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ee592f6e843c8a68a5170e0f9d66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.076195 29238 tablet_bootstrap.cc:654] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.078920 28430 tablet_service.cc:1467] Processing CreateTablet for tablet 233ee592f6e843c8a68a5170e0f9d66d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.079411 28430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ee592f6e843c8a68a5170e0f9d66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.080219 28298 tablet_service.cc:1467] Processing CreateTablet for tablet 233ee592f6e843c8a68a5170e0f9d66d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.080726 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ee592f6e843c8a68a5170e0f9d66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.085745 29238 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:42.085757 28166 tablet_service.cc:1467] Processing CreateTablet for tablet 77469d6a6610402ca96598bb695224d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.086061 29238 ts_tablet_manager.cc:1367] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20240503 14:15:42.086258 28166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77469d6a6610402ca96598bb695224d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.087124 29238 raft_consensus.cc:348] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.087535 29238 raft_consensus.cc:374] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.087664 29238 raft_consensus.cc:729] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.088019 29238 consensus_queue.cc:260] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.088879 29238 ts_tablet_manager.cc:1398] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:42.091600 29243 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:42.102511 29242 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:42.103379 29243 tablet_bootstrap.cc:654] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.104283 28035 tablet_service.cc:1467] Processing CreateTablet for tablet 77469d6a6610402ca96598bb695224d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.104564 28298 tablet_service.cc:1467] Processing CreateTablet for tablet 77469d6a6610402ca96598bb695224d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.104764 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77469d6a6610402ca96598bb695224d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.105034 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77469d6a6610402ca96598bb695224d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.112990 29238 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:42.113412 29242 tablet_bootstrap.cc:654] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.114593 29238 tablet_bootstrap.cc:654] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.116637 29242 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:42.116812 29242 ts_tablet_manager.cc:1367] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20240503 14:15:42.116730 28431 tablet_service.cc:1467] Processing CreateTablet for tablet 77469d6a6610402ca96598bb695224d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.117193 28431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77469d6a6610402ca96598bb695224d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.117724 27640 tablet_service.cc:1467] Processing CreateTablet for tablet 77469d6a6610402ca96598bb695224d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.117642 29242 raft_consensus.cc:348] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.117929 29242 raft_consensus.cc:374] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.118022 29242 raft_consensus.cc:729] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.118203 27640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77469d6a6610402ca96598bb695224d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.118772 27905 tablet_service.cc:1467] Processing CreateTablet for tablet 77469d6a6610402ca96598bb695224d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.119247 27905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77469d6a6610402ca96598bb695224d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.120239 29242 consensus_queue.cc:260] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.121121 29239 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:42.121785 27772 tablet_service.cc:1467] Processing CreateTablet for tablet 77469d6a6610402ca96598bb695224d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.122246 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77469d6a6610402ca96598bb695224d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.122860 29239 tablet_bootstrap.cc:654] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.123548 29247 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:42.124711 29242 ts_tablet_manager.cc:1398] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240503 14:15:42.126494 29247 tablet_bootstrap.cc:654] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.127254 28167 tablet_service.cc:1467] Processing CreateTablet for tablet bbb645fc3c5c422b9e3fabba8ce12873 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.127537 29238 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:42.127810 29238 ts_tablet_manager.cc:1367] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.007s
I20240503 14:15:42.128007 28167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb645fc3c5c422b9e3fabba8ce12873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.128486 27773 tablet_service.cc:1467] Processing CreateTablet for tablet bbb645fc3c5c422b9e3fabba8ce12873 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.128911 27773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb645fc3c5c422b9e3fabba8ce12873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.128831 29238 raft_consensus.cc:348] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.129276 29238 raft_consensus.cc:374] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.129424 29238 raft_consensus.cc:729] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.129786 29238 consensus_queue.cc:260] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.130558 29238 ts_tablet_manager.cc:1398] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:42.131299 28429 tablet_service.cc:1467] Processing CreateTablet for tablet bbb645fc3c5c422b9e3fabba8ce12873 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.131736 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb645fc3c5c422b9e3fabba8ce12873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.131760 28298 tablet_service.cc:1467] Processing CreateTablet for tablet bbb645fc3c5c422b9e3fabba8ce12873 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.132189 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb645fc3c5c422b9e3fabba8ce12873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.133608 29243 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:42.133821 29243 ts_tablet_manager.cc:1367] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.000s
I20240503 14:15:42.134653 29243 raft_consensus.cc:348] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.135568 29243 raft_consensus.cc:374] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.135708 29243 raft_consensus.cc:729] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.136010 29165 raft_consensus.cc:482] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:42.136152 29165 raft_consensus.cc:504] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.137566 29165 leader_election.cc:288] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:42.138285 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
W20240503 14:15:42.138904 27991 leader_election.cc:341] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:15:42.139742 27993 leader_election.cc:341] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605): Not found: Tablet not found: 77469d6a6610402ca96598bb695224d9
W20240503 14:15:42.139945 27993 leader_election.cc:341] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): Not found: Tablet not found: 77469d6a6610402ca96598bb695224d9
I20240503 14:15:42.140054 29242 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
W20240503 14:15:42.140147 27993 leader_election.cc:341] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119): Not found: Tablet not found: 77469d6a6610402ca96598bb695224d9
I20240503 14:15:42.140239 27993 leader_election.cc:302] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11; no voters: 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422
I20240503 14:15:42.141175 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:42.141433 29165 raft_consensus.cc:2726] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:15:42.142279 27992 leader_election.cc:341] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Not found: Tablet not found: 77469d6a6610402ca96598bb695224d9
W20240503 14:15:42.142478 27992 leader_election.cc:341] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:42.143127 29243 consensus_queue.cc:260] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.143991 29243 ts_tablet_manager.cc:1398] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:15:42.144376 29243 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:42.146975 29243 tablet_bootstrap.cc:654] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.149595 29242 tablet_bootstrap.cc:654] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.151489 29247 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:42.151717 29247 ts_tablet_manager.cc:1367] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240503 14:15:42.152649 29247 raft_consensus.cc:348] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.153015 29247 raft_consensus.cc:374] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.152974 28035 tablet_service.cc:1467] Processing CreateTablet for tablet bbb645fc3c5c422b9e3fabba8ce12873 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.153194 29247 raft_consensus.cc:729] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.153519 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb645fc3c5c422b9e3fabba8ce12873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.153620 29247 consensus_queue.cc:260] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.155109 29247 ts_tablet_manager.cc:1398] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:42.155436 29247 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:42.157986 29247 tablet_bootstrap.cc:654] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.158630 27640 tablet_service.cc:1467] Processing CreateTablet for tablet bbb645fc3c5c422b9e3fabba8ce12873 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.159139 27640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb645fc3c5c422b9e3fabba8ce12873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.158746 27905 tablet_service.cc:1467] Processing CreateTablet for tablet bbb645fc3c5c422b9e3fabba8ce12873 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.160207 27905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb645fc3c5c422b9e3fabba8ce12873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.166468 29250 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:42.168418 29250 tablet_bootstrap.cc:654] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.170104 29240 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:42.172067 29240 tablet_bootstrap.cc:654] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.172366 29239 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:42.172541 29239 ts_tablet_manager.cc:1367] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.000s
I20240503 14:15:42.173413 29239 raft_consensus.cc:348] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.173748 29239 raft_consensus.cc:374] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.173825 29239 raft_consensus.cc:729] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.174113 29239 consensus_queue.cc:260] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.175426 29239 ts_tablet_manager.cc:1398] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:42.175736 29239 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:42.177979 29238 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:42.179287 29239 tablet_bootstrap.cc:654] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.179651 29238 tablet_bootstrap.cc:654] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.187608 29242 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:42.187839 29242 ts_tablet_manager.cc:1367] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.004s
I20240503 14:15:42.188654 29242 raft_consensus.cc:348] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.188880 29239 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:42.188982 29242 raft_consensus.cc:374] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.189051 29242 raft_consensus.cc:729] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.189064 29239 ts_tablet_manager.cc:1367] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240503 14:15:42.189345 29242 consensus_queue.cc:260] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.190018 29242 ts_tablet_manager.cc:1398] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:42.190358 29242 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:42.191916 29242 tablet_bootstrap.cc:654] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.194510 29243 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:42.194803 29243 ts_tablet_manager.cc:1367] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.000s
I20240503 14:15:42.195302 29239 raft_consensus.cc:348] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.195878 29239 raft_consensus.cc:374] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.195998 29239 raft_consensus.cc:729] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.195672 29243 raft_consensus.cc:348] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.196224 29243 raft_consensus.cc:374] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.196345 29243 raft_consensus.cc:729] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.196290 29239 consensus_queue.cc:260] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.196830 29243 consensus_queue.cc:260] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.197132 29238 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:42.197327 29238 ts_tablet_manager.cc:1367] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20240503 14:15:42.197814 29243 ts_tablet_manager.cc:1398] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:42.198194 29243 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:42.198242 29238 raft_consensus.cc:348] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.198501 29238 raft_consensus.cc:374] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.198594 29238 raft_consensus.cc:729] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.198920 29238 consensus_queue.cc:260] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.200124 29243 tablet_bootstrap.cc:654] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.200634 29238 ts_tablet_manager.cc:1398] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:42.202636 29250 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:42.205080 29239 ts_tablet_manager.cc:1398] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20240503 14:15:42.205437 29239 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:42.206169 29250 ts_tablet_manager.cc:1367] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240503 14:15:42.206858 29240 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:42.207042 29240 ts_tablet_manager.cc:1367] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240503 14:15:42.207068 29239 tablet_bootstrap.cc:654] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.207214 29250 raft_consensus.cc:348] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.207572 29250 raft_consensus.cc:374] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.207763 29250 raft_consensus.cc:729] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.207836 29240 raft_consensus.cc:348] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.208106 29240 raft_consensus.cc:374] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.208199 29240 raft_consensus.cc:729] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.208177 29250 consensus_queue.cc:260] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.208427 29240 consensus_queue.cc:260] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.208925 29239 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:42.209066 29239 ts_tablet_manager.cc:1367] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:15:42.209070 29250 ts_tablet_manager.cc:1398] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:42.209390 29250 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:42.210203 29247 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:42.210220 29242 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:42.210388 29242 ts_tablet_manager.cc:1367] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20240503 14:15:42.210381 29247 ts_tablet_manager.cc:1367] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.004s
I20240503 14:15:42.210911 29250 tablet_bootstrap.cc:654] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.211158 29242 raft_consensus.cc:348] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.211414 29242 raft_consensus.cc:374] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.211505 29242 raft_consensus.cc:729] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.211759 29242 consensus_queue.cc:260] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.212620 29239 raft_consensus.cc:348] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.212911 29239 raft_consensus.cc:374] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.213021 29239 raft_consensus.cc:729] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.213361 29239 consensus_queue.cc:260] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.214630 29242 ts_tablet_manager.cc:1398] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:15:42.215268 29239 ts_tablet_manager.cc:1398] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:15:42.215936 29247 raft_consensus.cc:348] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.216226 29247 raft_consensus.cc:374] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.216280 29250 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:42.216296 29247 raft_consensus.cc:729] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.216440 29250 ts_tablet_manager.cc:1367] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240503 14:15:42.216585 29247 consensus_queue.cc:260] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.217170 29250 raft_consensus.cc:348] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.217470 29250 raft_consensus.cc:374] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.217552 29250 raft_consensus.cc:729] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.218003 29247 ts_tablet_manager.cc:1398] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20240503 14:15:42.217810 29250 consensus_queue.cc:260] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.218300 29247 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:42.218443 29250 ts_tablet_manager.cc:1398] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:42.218739 29250 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:42.219906 29247 tablet_bootstrap.cc:654] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.220224 29250 tablet_bootstrap.cc:654] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.221640 29240 ts_tablet_manager.cc:1398] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240503 14:15:42.221994 29240 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:42.223649 29240 tablet_bootstrap.cc:654] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.224161 29250 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:42.224321 29250 ts_tablet_manager.cc:1367] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240503 14:15:42.225122 29250 raft_consensus.cc:348] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.225426 29250 raft_consensus.cc:374] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.225514 29250 raft_consensus.cc:729] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.225800 29243 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:42.225780 29250 consensus_queue.cc:260] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.226079 29243 ts_tablet_manager.cc:1367] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.004s
I20240503 14:15:42.227059 29124 raft_consensus.cc:482] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:42.227301 29124 raft_consensus.cc:504] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.227367 29243 raft_consensus.cc:348] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.227666 29243 raft_consensus.cc:374] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.227757 29243 raft_consensus.cc:729] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.228070 29243 consensus_queue.cc:260] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.228731 29124 leader_election.cc:288] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:42.229008 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:42.229243 28450 raft_consensus.cc:2454] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:42.229588 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:42.229858 28057 raft_consensus.cc:2454] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:42.229868 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:42.230094 28187 raft_consensus.cc:2454] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:42.230165 29243 ts_tablet_manager.cc:1398] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:15:42.230115 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:42.230439 27925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
W20240503 14:15:42.231127 27598 leader_election.cc:341] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:42.231140 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:42.231346 28319 raft_consensus.cc:2454] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:42.231456 27598 leader_election.cc:302] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 74318b6270b64efaa79867167fc08bbb
I20240503 14:15:42.231657 29250 ts_tablet_manager.cc:1398] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20240503 14:15:42.231757 27598 leader_election.cc:341] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:42.232385 29124 raft_consensus.cc:2781] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:42.232477 29124 raft_consensus.cc:482] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:42.232571 29124 raft_consensus.cc:3037] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.234899 29124 raft_consensus.cc:504] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.235896 29124 leader_election.cc:288] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Requested vote from peers 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:42.236119 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:42.236380 28057 raft_consensus.cc:3037] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.236374 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:42.236670 28187 raft_consensus.cc:3037] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.237354 29240 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:42.237562 29240 ts_tablet_manager.cc:1367] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20240503 14:15:42.237926 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:42.238142 28319 raft_consensus.cc:3037] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.238377 29240 raft_consensus.cc:348] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.238703 29240 raft_consensus.cc:374] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.238780 29240 raft_consensus.cc:729] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.239403 29240 consensus_queue.cc:260] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.239876 29247 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:42.239876 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:42.240077 29240 ts_tablet_manager.cc:1398] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:42.240118 28450 raft_consensus.cc:3037] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.240402 29247 ts_tablet_manager.cc:1367] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20240503 14:15:42.240501 29240 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:42.241241 29247 raft_consensus.cc:348] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.241611 29247 raft_consensus.cc:374] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.241756 29247 raft_consensus.cc:729] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.242091 29247 consensus_queue.cc:260] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.242300 29240 tablet_bootstrap.cc:654] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.242795 28450 raft_consensus.cc:2454] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.242931 29247 ts_tablet_manager.cc:1398] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:42.243942 28187 raft_consensus.cc:2454] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.244460 28057 raft_consensus.cc:2454] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.246568 27598 leader_election.cc:302] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:42.246737 27925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:42.246959 27925 raft_consensus.cc:3037] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.247197 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb645fc3c5c422b9e3fabba8ce12873" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:42.247429 27794 raft_consensus.cc:3037] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.249486 27794 raft_consensus.cc:2454] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.249938 27925 raft_consensus.cc:2454] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.249995 29124 raft_consensus.cc:2781] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:42.250181 29124 raft_consensus.cc:686] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Becoming Leader. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Running, Role: LEADER
I20240503 14:15:42.250487 29124 consensus_queue.cc:237] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.253185 28319 raft_consensus.cc:2454] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.255257 27516 catalog_manager.cc:5484] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:42.257449 29240 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:42.257640 29240 ts_tablet_manager.cc:1367] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20240503 14:15:42.259018 29240 raft_consensus.cc:348] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.259248 29240 raft_consensus.cc:374] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.259335 29240 raft_consensus.cc:729] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.259531 29240 consensus_queue.cc:260] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.259996 29240 ts_tablet_manager.cc:1398] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:42.302311 29124 raft_consensus.cc:482] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:42.302501 29124 raft_consensus.cc:504] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.303746 29124 leader_election.cc:288] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:42.303886 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:42.304021 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:42.304116 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:42.304203 28057 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:42.304203 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:42.304266 28450 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:42.304349 28187 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:42.304687 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:42.304949 28319 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:42.304937 27598 leader_election.cc:302] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:42.304426 27794 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:42.305176 27925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:42.305441 27925 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:42.305243 29124 raft_consensus.cc:2781] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:42.305805 29124 raft_consensus.cc:482] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:42.305915 29124 raft_consensus.cc:3037] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.307721 28954 raft_consensus.cc:482] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:42.307881 28954 raft_consensus.cc:504] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.308140 29124 raft_consensus.cc:504] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.309105 28954 leader_election.cc:288] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:42.309311 29124 leader_election.cc:288] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Requested vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:42.309265 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:42.309477 28187 raft_consensus.cc:3037] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.309538 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:42.309716 27662 raft_consensus.cc:2454] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:42.310089 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:42.310192 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:42.310289 28319 raft_consensus.cc:3037] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.310396 28057 raft_consensus.cc:2454] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:42.310837 27925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:42.311133 27925 raft_consensus.cc:2454] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:42.311596 28256 leader_election.cc:302] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91; no voters:
I20240503 14:15:42.311769 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:42.311882 28954 raft_consensus.cc:2781] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:42.311980 28954 raft_consensus.cc:482] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:42.311995 28451 raft_consensus.cc:3037] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.312052 28954 raft_consensus.cc:3037] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.312389 28187 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.312693 28319 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.312809 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:42.313071 28450 raft_consensus.cc:2454] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:42.314352 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:42.314431 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:42.314571 28057 raft_consensus.cc:3037] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.314687 27793 raft_consensus.cc:3037] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.314956 28451 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.314899 28954 raft_consensus.cc:504] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.315403 27925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77469d6a6610402ca96598bb695224d9" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:42.315649 27925 raft_consensus.cc:3037] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.316226 28954 leader_election.cc:288] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 election: Requested vote from peers 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:42.317209 28057 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.317333 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:42.317539 28188 raft_consensus.cc:3037] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.317674 27598 leader_election.cc:302] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:42.317817 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:42.317940 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:42.318048 27926 raft_consensus.cc:3037] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.318071 29124 raft_consensus.cc:2781] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:42.318148 27662 raft_consensus.cc:3037] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.318519 29124 raft_consensus.cc:686] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Becoming Leader. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Running, Role: LEADER
I20240503 14:15:42.318883 29124 consensus_queue.cc:237] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:42.320079 28188 raft_consensus.cc:2454] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:42.320461 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:42.320724 27792 raft_consensus.cc:3037] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.320816 27926 raft_consensus.cc:2454] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:42.321151 27662 raft_consensus.cc:2454] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:42.321341 28056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:42.321558 28056 raft_consensus.cc:3037] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.323001 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:42.323602 27792 raft_consensus.cc:2454] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:42.323961 27925 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.324020 28056 raft_consensus.cc:2454] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:42.325500 28256 leader_election.cc:302] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91; no voters:
I20240503 14:15:42.325723 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:42.325954 28451 raft_consensus.cc:3037] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.326067 28954 raft_consensus.cc:2781] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:42.326167 27793 raft_consensus.cc:2454] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:42.313891 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "233ee592f6e843c8a68a5170e0f9d66d" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:42.326606 28187 raft_consensus.cc:2362] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1. Re-sending same reply.
I20240503 14:15:42.327075 28954 raft_consensus.cc:686] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Becoming Leader. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Running, Role: LEADER
I20240503 14:15:42.327684 28954 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:42.328786 28451 raft_consensus.cc:2454] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:42.330417 27521 catalog_manager.cc:5484] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:42.332044 27516 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6). New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:42.661085 29124 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.688227 29124 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:42.709717 29271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:42.710140 29271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:42.718020 29271 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:42.720873 29124 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.732262 29124 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.745983 29124 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.756578 28954 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.758765 29124 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.800081 29280 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 14:15:42.826684 28954 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.842396 29280 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.848030 28954 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.856405 29269 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.857036 29269 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.863205 29280 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.870821 28954 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.875222 29280 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.880965 29124 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.881350 29280 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:43.128319 27597 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36073, user_credentials={real_user=slave}} blocked reactor thread for 136979us
INSERT report
rows total: 2000
time total: 444.134 ms
time per row: 0.222067 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:15:44.295939 27513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51414:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:44.296701 27513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_3 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 14:15:44.305657 28298 tablet_service.cc:1467] Processing CreateTablet for tablet 29945558359248c6a8a4c68952cfcc81 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.305898 28035 tablet_service.cc:1467] Processing CreateTablet for tablet 29945558359248c6a8a4c68952cfcc81 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.306203 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29945558359248c6a8a4c68952cfcc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.306315 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29945558359248c6a8a4c68952cfcc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.310895 27640 tablet_service.cc:1467] Processing CreateTablet for tablet 29945558359248c6a8a4c68952cfcc81 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.311349 27640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29945558359248c6a8a4c68952cfcc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.311545 27905 tablet_service.cc:1467] Processing CreateTablet for tablet 29945558359248c6a8a4c68952cfcc81 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.311936 27905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29945558359248c6a8a4c68952cfcc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.311898 28167 tablet_service.cc:1467] Processing CreateTablet for tablet 05e093a194fc4500a9926c260ecbb308 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.312367 28167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e093a194fc4500a9926c260ecbb308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.313681 27642 tablet_service.cc:1467] Processing CreateTablet for tablet 05e093a194fc4500a9926c260ecbb308 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.314074 27642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e093a194fc4500a9926c260ecbb308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.317684 27641 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9e6daafc14437faff996cfb40f6dfb (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.317806 27774 tablet_service.cc:1467] Processing CreateTablet for tablet 29945558359248c6a8a4c68952cfcc81 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.318070 27641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9e6daafc14437faff996cfb40f6dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.318289 27774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29945558359248c6a8a4c68952cfcc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.321863 27772 tablet_service.cc:1467] Processing CreateTablet for tablet 05e093a194fc4500a9926c260ecbb308 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.321907 28166 tablet_service.cc:1467] Processing CreateTablet for tablet 29945558359248c6a8a4c68952cfcc81 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.322301 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e093a194fc4500a9926c260ecbb308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.322367 28166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29945558359248c6a8a4c68952cfcc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.325763 28431 tablet_service.cc:1467] Processing CreateTablet for tablet 29945558359248c6a8a4c68952cfcc81 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.325829 27906 tablet_service.cc:1467] Processing CreateTablet for tablet 05e093a194fc4500a9926c260ecbb308 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.326239 28431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29945558359248c6a8a4c68952cfcc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.326308 27906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e093a194fc4500a9926c260ecbb308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.329690 28430 tablet_service.cc:1467] Processing CreateTablet for tablet 05e093a194fc4500a9926c260ecbb308 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.329757 27904 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9e6daafc14437faff996cfb40f6dfb (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.330121 28430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e093a194fc4500a9926c260ecbb308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.330215 27904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9e6daafc14437faff996cfb40f6dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.333794 28168 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9e6daafc14437faff996cfb40f6dfb (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.334244 28168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9e6daafc14437faff996cfb40f6dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.335505 28036 tablet_service.cc:1467] Processing CreateTablet for tablet 05e093a194fc4500a9926c260ecbb308 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.335983 28036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e093a194fc4500a9926c260ecbb308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.337507 29385 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:44.337672 28429 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9e6daafc14437faff996cfb40f6dfb (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.338075 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9e6daafc14437faff996cfb40f6dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.339203 29385 tablet_bootstrap.cc:654] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.339696 28037 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9e6daafc14437faff996cfb40f6dfb (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.340178 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9e6daafc14437faff996cfb40f6dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.341964 27773 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9e6daafc14437faff996cfb40f6dfb (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.342444 27773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9e6daafc14437faff996cfb40f6dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.350952 28299 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9e6daafc14437faff996cfb40f6dfb (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.351498 28299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9e6daafc14437faff996cfb40f6dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.345557 29385 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:44.359692 29385 ts_tablet_manager.cc:1367] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.004s
I20240503 14:15:44.360522 29385 raft_consensus.cc:348] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.363451 29385 raft_consensus.cc:374] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.363559 29385 raft_consensus.cc:729] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.364886 29385 consensus_queue.cc:260] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.366339 29385 ts_tablet_manager.cc:1398] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:15:44.395000 29385 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:44.396766 29385 tablet_bootstrap.cc:654] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.400236 28298 tablet_service.cc:1467] Processing CreateTablet for tablet 05e093a194fc4500a9926c260ecbb308 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.400902 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e093a194fc4500a9926c260ecbb308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.411936 29392 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:44.412647 29393 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:44.412715 29389 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:44.414105 29390 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:44.414767 29389 tablet_bootstrap.cc:654] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.416868 29392 tablet_bootstrap.cc:654] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
W20240503 14:15:44.417512 28256 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:36119, user_credentials={real_user=slave}} blocked reactor thread for 44535.8us
I20240503 14:15:44.423463 29384 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:44.424463 29386 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:44.425503 29390 tablet_bootstrap.cc:654] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.426105 29386 tablet_bootstrap.cc:654] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.427192 29384 tablet_bootstrap.cc:654] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.427595 29389 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:44.427862 29393 tablet_bootstrap.cc:654] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.428920 29389 ts_tablet_manager.cc:1367] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240503 14:15:44.432003 29389 raft_consensus.cc:348] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.432943 29389 raft_consensus.cc:374] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.433048 29389 raft_consensus.cc:729] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.433843 29390 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:44.434020 29390 ts_tablet_manager.cc:1367] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20240503 14:15:44.434393 29389 consensus_queue.cc:260] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.434708 29386 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:44.434921 29386 ts_tablet_manager.cc:1367] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 14:15:44.434829 29390 raft_consensus.cc:348] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.435113 29390 raft_consensus.cc:374] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.435225 29390 raft_consensus.cc:729] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.435369 29384 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:44.435546 29384 ts_tablet_manager.cc:1367] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240503 14:15:44.435501 29390 consensus_queue.cc:260] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.436090 29386 raft_consensus.cc:348] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.436429 29386 raft_consensus.cc:374] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.436564 29386 raft_consensus.cc:729] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.436911 29390 ts_tablet_manager.cc:1398] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:44.436899 29386 consensus_queue.cc:260] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.437237 29390 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:44.438580 29386 ts_tablet_manager.cc:1398] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:15:44.438925 29386 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:44.438983 29384 raft_consensus.cc:348] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.440990 29384 raft_consensus.cc:374] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.441128 29384 raft_consensus.cc:729] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.441848 29386 tablet_bootstrap.cc:654] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.441900 29385 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:44.442118 29385 ts_tablet_manager.cc:1367] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.003s
I20240503 14:15:44.442893 29385 raft_consensus.cc:348] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.443172 29385 raft_consensus.cc:374] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.443248 29385 raft_consensus.cc:729] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.443475 29385 consensus_queue.cc:260] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.444077 29385 ts_tablet_manager.cc:1398] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:44.444430 29385 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:44.444999 29390 tablet_bootstrap.cc:654] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.445477 29393 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:44.445657 29393 ts_tablet_manager.cc:1367] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20240503 14:15:44.446467 29393 raft_consensus.cc:348] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.446725 29385 tablet_bootstrap.cc:654] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.446748 29393 raft_consensus.cc:374] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.446842 29393 raft_consensus.cc:729] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.447744 29384 consensus_queue.cc:260] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.449333 29384 ts_tablet_manager.cc:1398] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240503 14:15:44.450110 29384 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:44.450678 29392 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:44.450927 29392 ts_tablet_manager.cc:1367] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.005s
I20240503 14:15:44.450960 29386 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:44.451215 29386 ts_tablet_manager.cc:1367] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240503 14:15:44.451772 29392 raft_consensus.cc:348] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.452039 29386 raft_consensus.cc:348] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.452407 29386 raft_consensus.cc:374] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.452317 29393 consensus_queue.cc:260] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.452531 29386 raft_consensus.cc:729] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.452929 29386 consensus_queue.cc:260] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.453130 29393 ts_tablet_manager.cc:1398] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:15:44.454442 29386 ts_tablet_manager.cc:1398] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:44.454854 29386 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:44.454917 29384 tablet_bootstrap.cc:654] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.456508 29386 tablet_bootstrap.cc:654] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.458557 29392 raft_consensus.cc:374] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.458745 29392 raft_consensus.cc:729] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.460414 29392 consensus_queue.cc:260] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.460651 29390 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:44.460904 29390 ts_tablet_manager.cc:1367] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20240503 14:15:44.461160 29393 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:44.462774 29384 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:44.462962 29384 ts_tablet_manager.cc:1367] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 14:15:44.463268 29390 raft_consensus.cc:348] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.463603 29390 raft_consensus.cc:374] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.463680 29385 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:44.463696 29390 raft_consensus.cc:729] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.463852 29385 ts_tablet_manager.cc:1367] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.006s
I20240503 14:15:44.464015 29390 consensus_queue.cc:260] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.463689 29384 raft_consensus.cc:348] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.464323 29384 raft_consensus.cc:374] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.464406 29384 raft_consensus.cc:729] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.464618 29385 raft_consensus.cc:348] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.464897 29385 raft_consensus.cc:374] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.464987 29385 raft_consensus.cc:729] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.465407 29390 ts_tablet_manager.cc:1398] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:44.465240 29385 consensus_queue.cc:260] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.465746 29390 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:44.465863 29385 ts_tablet_manager.cc:1398] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:44.466284 29389 ts_tablet_manager.cc:1398] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20240503 14:15:44.466692 29389 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:44.467152 29393 tablet_bootstrap.cc:654] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.468547 29389 tablet_bootstrap.cc:654] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.469429 29392 ts_tablet_manager.cc:1398] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 14:15:44.464706 29384 consensus_queue.cc:260] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.471741 29384 ts_tablet_manager.cc:1398] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 14:15:44.472100 29384 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:44.474411 29392 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:44.476282 29384 tablet_bootstrap.cc:654] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.476300 29392 tablet_bootstrap.cc:654] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.477048 29390 tablet_bootstrap.cc:654] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.482263 29392 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:44.483042 29393 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:44.483249 29393 ts_tablet_manager.cc:1367] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.009s
I20240503 14:15:44.483287 29389 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:44.483418 29392 ts_tablet_manager.cc:1367] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20240503 14:15:44.483458 29389 ts_tablet_manager.cc:1367] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240503 14:15:44.484097 29393 raft_consensus.cc:348] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.484126 29389 raft_consensus.cc:348] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.484196 29392 raft_consensus.cc:348] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.484405 29389 raft_consensus.cc:374] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.484405 29393 raft_consensus.cc:374] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.484447 29392 raft_consensus.cc:374] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.484485 29389 raft_consensus.cc:729] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.484479 29393 raft_consensus.cc:729] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.484511 29392 raft_consensus.cc:729] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.484762 29392 consensus_queue.cc:260] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.484774 29393 consensus_queue.cc:260] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.485357 29392 ts_tablet_manager.cc:1398] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:44.485415 29393 ts_tablet_manager.cc:1398] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:44.484773 29389 consensus_queue.cc:260] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.485780 29393 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:44.486263 29392 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:44.487216 29389 ts_tablet_manager.cc:1398] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:15:44.487864 29392 tablet_bootstrap.cc:654] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.487895 29389 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:44.489833 29393 tablet_bootstrap.cc:654] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.489888 29386 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:44.490058 29386 ts_tablet_manager.cc:1367] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.000s
I20240503 14:15:44.490000 29389 tablet_bootstrap.cc:654] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.490898 29386 raft_consensus.cc:348] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.491219 29386 raft_consensus.cc:374] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.491313 29386 raft_consensus.cc:729] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.491562 29386 consensus_queue.cc:260] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.492182 29386 ts_tablet_manager.cc:1398] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:44.492539 29391 raft_consensus.cc:482] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:44.492728 29391 raft_consensus.cc:504] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.494073 29391 leader_election.cc:288] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:44.495494 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:44.495584 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:44.495741 28187 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:44.496083 28055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:44.496315 28055 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:44.496809 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:44.497045 27924 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:44.497505 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:44.497572 28256 leader_election.cc:302] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:44.497757 28451 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:44.498456 29391 raft_consensus.cc:2781] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:44.498576 29391 raft_consensus.cc:482] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 14:15:44.498690 28256 leader_election.cc:341] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:44.498879 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:44.499007 29389 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:44.499173 29389 ts_tablet_manager.cc:1367] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 14:15:44.499137 27662 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:44.498679 29391 raft_consensus.cc:3037] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.500005 29389 raft_consensus.cc:348] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.500275 29389 raft_consensus.cc:374] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.500361 29389 raft_consensus.cc:729] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.500624 29389 consensus_queue.cc:260] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.501252 29389 ts_tablet_manager.cc:1398] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:15:44.502564 29391 raft_consensus.cc:504] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.504634 28055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:44.504832 28055 raft_consensus.cc:3037] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.504981 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:44.505218 28187 raft_consensus.cc:3037] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.505223 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:44.505481 28451 raft_consensus.cc:3037] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.506003 29384 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:44.506384 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:44.506464 29384 ts_tablet_manager.cc:1367] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.006s
I20240503 14:15:44.506605 27662 raft_consensus.cc:3037] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.506707 29390 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:44.506843 29390 ts_tablet_manager.cc:1367] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.002s
I20240503 14:15:44.507413 29384 raft_consensus.cc:348] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.507678 29384 raft_consensus.cc:374] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.507764 29384 raft_consensus.cc:729] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.507623 29390 raft_consensus.cc:348] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.507894 29390 raft_consensus.cc:374] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.507989 29390 raft_consensus.cc:729] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.508010 28055 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:44.508136 29384 consensus_queue.cc:260] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.508239 29390 consensus_queue.cc:260] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.508875 29390 ts_tablet_manager.cc:1398] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:44.509395 29391 leader_election.cc:288] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 election: Requested vote from peers 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:44.509420 27662 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:44.510337 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:44.510464 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e093a194fc4500a9926c260ecbb308" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:44.510546 27924 raft_consensus.cc:3037] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.510677 27792 raft_consensus.cc:3037] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.512512 27792 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:44.513135 29384 ts_tablet_manager.cc:1398] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:15:44.513262 27924 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:44.517496 28256 leader_election.cc:302] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91; no voters:
I20240503 14:15:44.517944 29391 raft_consensus.cc:2781] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:44.518074 29391 raft_consensus.cc:686] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Becoming Leader. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Running, Role: LEADER
I20240503 14:15:44.518604 29391 consensus_queue.cc:237] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.523267 29392 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:44.523844 28187 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:44.524108 28451 raft_consensus.cc:2454] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:44.523424 27513 catalog_manager.cc:5484] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6). New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:44.525133 29393 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:44.525342 29393 ts_tablet_manager.cc:1367] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
I20240503 14:15:44.526280 29393 raft_consensus.cc:348] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.526605 29393 raft_consensus.cc:374] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.526695 29393 raft_consensus.cc:729] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.527067 29393 consensus_queue.cc:260] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.524158 29392 ts_tablet_manager.cc:1367] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.002s
I20240503 14:15:44.528584 29393 ts_tablet_manager.cc:1398] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:44.528779 29392 raft_consensus.cc:348] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.529119 29392 raft_consensus.cc:374] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.529212 29392 raft_consensus.cc:729] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.529505 29392 consensus_queue.cc:260] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.530287 29392 ts_tablet_manager.cc:1398] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:44.582799 29375 raft_consensus.cc:482] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:44.583009 29375 raft_consensus.cc:504] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.584379 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:44.584632 28187 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:44.585362 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:44.585386 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:44.585443 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:44.585580 27662 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:44.585637 27792 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:44.585696 27924 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:44.584259 29375 leader_election.cc:288] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:44.586354 27993 leader_election.cc:302] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad; no voters:
I20240503 14:15:44.586477 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:44.586699 28320 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:44.586753 29375 raft_consensus.cc:2781] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:44.586861 29375 raft_consensus.cc:482] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:44.586943 29375 raft_consensus.cc:3037] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.587378 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:44.587627 28451 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:44.589423 29375 raft_consensus.cc:504] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.590672 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:44.590690 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:44.590884 28187 raft_consensus.cc:3037] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.590891 28320 raft_consensus.cc:3037] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.593015 29375 leader_election.cc:288] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 election: Requested vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:44.593420 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:44.593600 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:44.593678 28451 raft_consensus.cc:3037] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.593642 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:44.593822 27924 raft_consensus.cc:3037] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.593873 27792 raft_consensus.cc:3037] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.596535 27924 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:44.596535 28451 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:44.597783 27792 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:44.597806 28320 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:44.598253 27992 leader_election.cc:302] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 9231e715a4e342a197d6c5456ba6b399, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91; no voters:
I20240503 14:15:44.598662 29375 raft_consensus.cc:2781] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:44.598809 29375 raft_consensus.cc:686] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Becoming Leader. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Running, Role: LEADER
I20240503 14:15:44.599299 28187 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:44.598615 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9e6daafc14437faff996cfb40f6dfb" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:44.599606 27662 raft_consensus.cc:3037] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.599200 29375 consensus_queue.cc:237] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.602741 27662 raft_consensus.cc:2454] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:44.605800 27513 catalog_manager.cc:5484] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 078f8eff84c94dde9705ad1330922a11 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:44.758984 29401 raft_consensus.cc:482] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:44.759253 29401 raft_consensus.cc:504] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.761628 29401 leader_election.cc:288] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:44.765414 29048 raft_consensus.cc:482] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:44.765686 29048 raft_consensus.cc:504] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.767192 29048 leader_election.cc:288] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:44.768031 28055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:44.768368 28055 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:44.768394 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:44.768661 27924 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:44.769603 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:44.769863 27792 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:44.771036 28124 leader_election.cc:302] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:44.771426 29048 raft_consensus.cc:2781] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:44.771553 29048 raft_consensus.cc:482] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:44.771639 29048 raft_consensus.cc:3037] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.774854 29048 raft_consensus.cc:504] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.776705 28055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:44.777004 28055 raft_consensus.cc:3037] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.779008 29048 leader_election.cc:288] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 election: Requested vote from peers acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:44.779466 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:44.779528 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:44.779728 27792 raft_consensus.cc:3037] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.779748 27924 raft_consensus.cc:3037] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.780297 28055 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:44.782368 27924 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:44.785995 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:44.786306 27662 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:44.786901 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:44.787184 28320 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:44.788089 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:44.788393 28451 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
I20240503 14:15:44.789363 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:44.789600 28320 raft_consensus.cc:3037] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.791092 28055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:44.791373 28055 raft_consensus.cc:2379] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74318b6270b64efaa79867167fc08bbb in current term 1: Already voted for candidate c91f285b8f4f4a27bcdd0ada303c1422 in this term.
I20240503 14:15:44.792877 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:44.793140 27662 raft_consensus.cc:3037] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.795588 28320 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:44.796110 27662 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:44.796463 28123 leader_election.cc:302] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:44.796649 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:44.796831 29048 raft_consensus.cc:2781] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:44.796888 28451 raft_consensus.cc:3037] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.800328 28451 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:44.803490 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:44.803835 27924 raft_consensus.cc:2379] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74318b6270b64efaa79867167fc08bbb in current term 1: Already voted for candidate c91f285b8f4f4a27bcdd0ada303c1422 in this term.
I20240503 14:15:44.803997 29048 raft_consensus.cc:686] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Becoming Leader. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Running, Role: LEADER
I20240503 14:15:44.804539 29048 consensus_queue.cc:237] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:44.804977 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:44.805251 28451 raft_consensus.cc:2379] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74318b6270b64efaa79867167fc08bbb in current term 1: Already voted for candidate c91f285b8f4f4a27bcdd0ada303c1422 in this term.
I20240503 14:15:44.805266 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:44.806107 27728 leader_election.cc:302] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 74318b6270b64efaa79867167fc08bbb; no voters: 078f8eff84c94dde9705ad1330922a11, 9231e715a4e342a197d6c5456ba6b399, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422
I20240503 14:15:44.807801 27792 raft_consensus.cc:2454] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 1.
I20240503 14:15:44.808171 29401 raft_consensus.cc:2726] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:44.809337 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:44.809674 28320 raft_consensus.cc:2379] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74318b6270b64efaa79867167fc08bbb in current term 1: Already voted for candidate c91f285b8f4f4a27bcdd0ada303c1422 in this term.
I20240503 14:15:44.812206 27513 catalog_manager.cc:5484] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: term changed from 0 to 1, leader changed from <none> to c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5). New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:44.815402 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29945558359248c6a8a4c68952cfcc81" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:44.815716 27662 raft_consensus.cc:2379] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74318b6270b64efaa79867167fc08bbb in current term 1: Already voted for candidate c91f285b8f4f4a27bcdd0ada303c1422 in this term.
I20240503 14:15:44.898178 28954 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:44.899003 28954 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:44.940405 28954 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:44.943614 28954 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:44.963894 29437 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:44.978147 29437 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.032275 29375 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.032871 29388 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.051574 29375 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.053197 29375 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.068121 29388 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.068903 29388 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:45.229557 29430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:45.229902 29430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:45.238103 29430 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 14:15:45.287665 29427 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.289983 29427 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.308631 29427 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.327061 29427 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.329453 29427 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.339459 29427 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:45.409891 28123 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:35261, user_credentials={real_user=slave}} blocked reactor thread for 65260.9us
W20240503 14:15:45.562007 28255 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:35261, user_credentials={real_user=slave}} blocked reactor thread for 78975.7us
W20240503 14:15:45.562558 28256 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:38859, user_credentials={real_user=slave}} blocked reactor thread for 80740.3us
INSERT report
rows total: 2000
time total: 582.494 ms
time per row: 0.291247 ms
I20240503 14:15:45.982153 27514 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51414:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:45.982926 27514 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_4 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 14:15:45.992272 27642 tablet_service.cc:1467] Processing CreateTablet for tablet d84acfe4f36f4193970fd2230229a41f (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:45.992822 27642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84acfe4f36f4193970fd2230229a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:45.992893 28167 tablet_service.cc:1467] Processing CreateTablet for tablet d84acfe4f36f4193970fd2230229a41f (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:45.992980 27773 tablet_service.cc:1467] Processing CreateTablet for tablet d84acfe4f36f4193970fd2230229a41f (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:45.993371 28167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84acfe4f36f4193970fd2230229a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:45.993476 27773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84acfe4f36f4193970fd2230229a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:45.997840 27905 tablet_service.cc:1467] Processing CreateTablet for tablet d84acfe4f36f4193970fd2230229a41f (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:45.998311 29535 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:45.998327 27905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84acfe4f36f4193970fd2230229a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:45.999310 28035 tablet_service.cc:1467] Processing CreateTablet for tablet d84acfe4f36f4193970fd2230229a41f (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:45.999764 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84acfe4f36f4193970fd2230229a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.000082 29535 tablet_bootstrap.cc:654] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.002291 28429 tablet_service.cc:1467] Processing CreateTablet for tablet d84acfe4f36f4193970fd2230229a41f (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.002749 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84acfe4f36f4193970fd2230229a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.004384 28036 tablet_service.cc:1467] Processing CreateTablet for tablet 4c23af4dc19c412685927c70cbee7124 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.004906 28036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23af4dc19c412685927c70cbee7124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.004837 27772 tablet_service.cc:1467] Processing CreateTablet for tablet 4c23af4dc19c412685927c70cbee7124 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.005265 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23af4dc19c412685927c70cbee7124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.005707 28431 tablet_service.cc:1467] Processing CreateTablet for tablet 4c23af4dc19c412685927c70cbee7124 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.006088 28431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23af4dc19c412685927c70cbee7124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.011374 28298 tablet_service.cc:1467] Processing CreateTablet for tablet d84acfe4f36f4193970fd2230229a41f (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.011869 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84acfe4f36f4193970fd2230229a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.012990 27906 tablet_service.cc:1467] Processing CreateTablet for tablet 4c23af4dc19c412685927c70cbee7124 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.013484 27906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23af4dc19c412685927c70cbee7124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.013706 28299 tablet_service.cc:1467] Processing CreateTablet for tablet 4c23af4dc19c412685927c70cbee7124 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.014076 28299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23af4dc19c412685927c70cbee7124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.015101 28167 tablet_service.cc:1467] Processing CreateTablet for tablet 4c23af4dc19c412685927c70cbee7124 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.015597 28167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23af4dc19c412685927c70cbee7124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.020578 27640 tablet_service.cc:1467] Processing CreateTablet for tablet 4c23af4dc19c412685927c70cbee7124 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.021068 27640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23af4dc19c412685927c70cbee7124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.025676 29539 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:46.027386 28300 tablet_service.cc:1467] Processing CreateTablet for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.027877 28300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b7e707cec54bc8bc5e649c3b37dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.028331 29537 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:46.028264 27641 tablet_service.cc:1467] Processing CreateTablet for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.028672 27641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b7e707cec54bc8bc5e649c3b37dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.029881 29539 tablet_bootstrap.cc:654] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.032027 29537 tablet_bootstrap.cc:654] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.036351 29543 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:46.037132 27906 tablet_service.cc:1467] Processing CreateTablet for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.037732 27906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b7e707cec54bc8bc5e649c3b37dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.037837 28036 tablet_service.cc:1467] Processing CreateTablet for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.038316 28036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b7e707cec54bc8bc5e649c3b37dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.041213 27773 tablet_service.cc:1467] Processing CreateTablet for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.041740 27773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b7e707cec54bc8bc5e649c3b37dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.044780 28167 tablet_service.cc:1467] Processing CreateTablet for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.045387 28167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b7e707cec54bc8bc5e649c3b37dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.047592 29543 tablet_bootstrap.cc:654] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.048198 29538 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:46.056425 29538 tablet_bootstrap.cc:654] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.056708 29542 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:46.058641 29542 tablet_bootstrap.cc:654] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.059506 28430 tablet_service.cc:1467] Processing CreateTablet for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.060142 29539 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:46.060153 28430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b7e707cec54bc8bc5e649c3b37dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.060418 29539 ts_tablet_manager.cc:1367] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.000s
I20240503 14:15:46.061407 29539 raft_consensus.cc:348] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.061870 29539 raft_consensus.cc:374] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.062021 29539 raft_consensus.cc:729] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.062374 29539 consensus_queue.cc:260] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.063246 29539 ts_tablet_manager.cc:1398] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:46.063592 29539 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:46.065138 29544 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:46.065258 29539 tablet_bootstrap.cc:654] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.070678 29544 tablet_bootstrap.cc:654] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.071113 29537 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:46.071285 29537 ts_tablet_manager.cc:1367] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.003s
I20240503 14:15:46.072183 29537 raft_consensus.cc:348] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.072507 29537 raft_consensus.cc:374] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.072588 29537 raft_consensus.cc:729] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.072886 29537 consensus_queue.cc:260] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.074298 29539 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:46.074556 29539 ts_tablet_manager.cc:1367] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:15:46.076391 29539 raft_consensus.cc:348] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.076716 29539 raft_consensus.cc:374] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.076795 29539 raft_consensus.cc:729] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.077303 29539 consensus_queue.cc:260] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.078055 29539 ts_tablet_manager.cc:1398] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:46.079363 29539 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:46.081059 29539 tablet_bootstrap.cc:654] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.083431 29535 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:46.083603 29535 ts_tablet_manager.cc:1367] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.085s user 0.038s sys 0.030s
I20240503 14:15:46.083804 29543 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:46.083961 29543 ts_tablet_manager.cc:1367] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240503 14:15:46.084453 29535 raft_consensus.cc:348] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.084756 29535 raft_consensus.cc:374] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.084847 29535 raft_consensus.cc:729] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.085117 29535 consensus_queue.cc:260] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.084728 29543 raft_consensus.cc:348] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.085896 29543 raft_consensus.cc:374] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.086009 29543 raft_consensus.cc:729] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.086247 29543 consensus_queue.cc:260] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.086514 29535 ts_tablet_manager.cc:1398] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:46.086879 29543 ts_tablet_manager.cc:1398] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:46.087188 29543 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:46.088382 29535 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:46.088914 29543 tablet_bootstrap.cc:654] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.089989 29535 tablet_bootstrap.cc:654] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.090767 29538 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:46.090955 29538 ts_tablet_manager.cc:1367] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.004s
I20240503 14:15:46.091852 29538 raft_consensus.cc:348] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.092149 29538 raft_consensus.cc:374] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.092226 29538 raft_consensus.cc:729] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.092504 29538 consensus_queue.cc:260] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.094075 29538 ts_tablet_manager.cc:1398] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:15:46.094404 29538 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:46.109725 29537 ts_tablet_manager.cc:1398] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.038s user 0.004s sys 0.004s
I20240503 14:15:46.110132 29547 raft_consensus.cc:482] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:46.110319 29547 raft_consensus.cc:504] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.111927 29538 tablet_bootstrap.cc:654] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.112301 29543 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:46.112488 29543 ts_tablet_manager.cc:1367] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240503 14:15:46.113139 29537 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:46.113348 29543 raft_consensus.cc:348] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.113668 29543 raft_consensus.cc:374] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.113741 29543 raft_consensus.cc:729] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.114001 29543 consensus_queue.cc:260] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.114693 29543 ts_tablet_manager.cc:1398] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:46.115027 29543 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:46.115409 29542 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:46.115571 29542 ts_tablet_manager.cc:1367] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.008s
I20240503 14:15:46.116379 29542 raft_consensus.cc:348] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.116667 29542 raft_consensus.cc:374] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.116755 29542 raft_consensus.cc:729] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.116988 29542 consensus_queue.cc:260] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.117821 29542 ts_tablet_manager.cc:1398] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:46.118163 29542 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:46.119776 29542 tablet_bootstrap.cc:654] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.120438 29543 tablet_bootstrap.cc:654] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.121817 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:46.121881 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:46.122124 27924 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 0.
W20240503 14:15:46.122643 27729 leader_election.cc:341] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:46.123991 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:46.124140 29544 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:46.124274 28188 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 0.
I20240503 14:15:46.124361 29544 ts_tablet_manager.cc:1367] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.000s
I20240503 14:15:46.125193 29544 raft_consensus.cc:348] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.125502 29544 raft_consensus.cc:374] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.125595 29544 raft_consensus.cc:729] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.125717 29539 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:46.125875 29539 ts_tablet_manager.cc:1367] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240503 14:15:46.125856 29544 consensus_queue.cc:260] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.127655 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:46.127941 29544 ts_tablet_manager.cc:1398] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:46.128247 29544 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
W20240503 14:15:46.128504 27730 leader_election.cc:341] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:46.129082 29535 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:46.129316 29535 ts_tablet_manager.cc:1367] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.004s
I20240503 14:15:46.129387 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:46.129626 29537 tablet_bootstrap.cc:654] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.129616 28451 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 0.
I20240503 14:15:46.130084 27728 leader_election.cc:302] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18
I20240503 14:15:46.130362 29547 leader_election.cc:288] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:46.130540 29547 raft_consensus.cc:2781] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:46.130635 29547 raft_consensus.cc:482] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:46.130714 29547 raft_consensus.cc:3037] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.130551 29535 raft_consensus.cc:348] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.130872 29535 raft_consensus.cc:374] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.130950 29535 raft_consensus.cc:729] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.131518 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
W20240503 14:15:46.132741 27729 leader_election.cc:341] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:46.126631 29539 raft_consensus.cc:348] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.133854 29539 raft_consensus.cc:374] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.133960 29539 raft_consensus.cc:729] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.134555 29535 consensus_queue.cc:260] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.135049 29542 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:46.135236 29542 ts_tablet_manager.cc:1367] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240503 14:15:46.135747 29543 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:46.135895 29543 ts_tablet_manager.cc:1367] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240503 14:15:46.135996 29542 raft_consensus.cc:348] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.136273 29542 raft_consensus.cc:374] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.136345 29542 raft_consensus.cc:729] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.136627 29542 consensus_queue.cc:260] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.137400 29542 ts_tablet_manager.cc:1398] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:46.137748 29542 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:46.139430 29542 tablet_bootstrap.cc:654] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.141206 29544 tablet_bootstrap.cc:654] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.141888 29535 ts_tablet_manager.cc:1398] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 14:15:46.142241 29535 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:46.143210 29538 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:46.143512 29538 ts_tablet_manager.cc:1367] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.004s
I20240503 14:15:46.143837 29535 tablet_bootstrap.cc:654] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.144356 29538 raft_consensus.cc:348] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.144873 29538 raft_consensus.cc:374] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.144961 29538 raft_consensus.cc:729] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.145237 29538 consensus_queue.cc:260] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.146085 29543 raft_consensus.cc:348] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.146391 29543 raft_consensus.cc:374] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.146497 29543 raft_consensus.cc:729] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.146723 29543 consensus_queue.cc:260] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.147316 29543 ts_tablet_manager.cc:1398] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 14:15:46.146600 29538 ts_tablet_manager.cc:1398] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:46.147747 29538 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:46.149403 29538 tablet_bootstrap.cc:654] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.150357 29539 consensus_queue.cc:260] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.151854 29539 ts_tablet_manager.cc:1398] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240503 14:15:46.154737 29542 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:46.154997 29542 ts_tablet_manager.cc:1367] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 14:15:46.155122 29391 raft_consensus.cc:482] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:46.155329 29391 raft_consensus.cc:504] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.155858 29542 raft_consensus.cc:348] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.156131 29542 raft_consensus.cc:374] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.156229 29542 raft_consensus.cc:729] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.156527 29391 leader_election.cc:288] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:46.156603 29547 raft_consensus.cc:504] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.158120 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:46.158349 27924 raft_consensus.cc:3037] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.158386 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:46.158635 27661 raft_consensus.cc:2454] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:46.159724 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:46.159909 29535 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:46.159940 28188 raft_consensus.cc:2454] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:46.160075 29535 ts_tablet_manager.cc:1367] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240503 14:15:46.160861 29535 raft_consensus.cc:348] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.161123 29535 raft_consensus.cc:374] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.161095 27924 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.161211 29535 raft_consensus.cc:729] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.161477 29535 consensus_queue.cc:260] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.161823 27925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:46.162029 27925 raft_consensus.cc:2454] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:46.162091 29535 ts_tablet_manager.cc:1398] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:46.162348 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:46.162559 28057 raft_consensus.cc:2454] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 0.
I20240503 14:15:46.162917 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:46.162891 29542 consensus_queue.cc:260] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.163535 29542 ts_tablet_manager.cc:1398] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 14:15:46.164069 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:46.164546 29537 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:46.164723 29537 ts_tablet_manager.cc:1367] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20240503 14:15:46.164817 28256 leader_election.cc:302] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91; no voters:
I20240503 14:15:46.165303 28954 raft_consensus.cc:2781] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:46.165418 28954 raft_consensus.cc:482] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:46.165508 28954 raft_consensus.cc:3037] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:15:46.165634 28254 leader_election.cc:341] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:46.165540 29537 raft_consensus.cc:348] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.165802 29537 raft_consensus.cc:374] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.165868 29537 raft_consensus.cc:729] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.166088 29537 consensus_queue.cc:260] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.166672 29537 ts_tablet_manager.cc:1398] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:46.166997 29537 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:46.167241 29494 raft_consensus.cc:482] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:46.167407 29494 raft_consensus.cc:504] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.168177 28954 raft_consensus.cc:504] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.168552 29537 tablet_bootstrap.cc:654] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.169874 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:46.170092 28451 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:46.170559 29547 leader_election.cc:288] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 election: Requested vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:46.170634 28954 leader_election.cc:288] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 election: Requested vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:46.170775 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:46.171239 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:46.171445 27661 raft_consensus.cc:3037] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.171939 29544 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:46.172113 29544 ts_tablet_manager.cc:1367] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.003s
I20240503 14:15:46.173055 29544 raft_consensus.cc:348] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.173425 29544 raft_consensus.cc:374] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.173506 29544 raft_consensus.cc:729] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.173753 29544 consensus_queue.cc:260] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.174407 27661 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.175354 29544 ts_tablet_manager.cc:1398] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:46.175493 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:46.175714 29544 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:46.175737 28320 raft_consensus.cc:3037] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.177531 29544 tablet_bootstrap.cc:654] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.178550 28320 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.179673 27729 leader_election.cc:302] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91; no voters:
I20240503 14:15:46.180055 29547 raft_consensus.cc:2781] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:46.180300 29547 raft_consensus.cc:686] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Becoming Leader. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Running, Role: LEADER
I20240503 14:15:46.180914 29547 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.181334 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:46.181557 28188 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:46.182142 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:46.182399 27924 raft_consensus.cc:2379] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7046cbd82d634976b53fe0cef5788f18 in current term 1: Already voted for candidate 74318b6270b64efaa79867167fc08bbb in this term.
I20240503 14:15:46.183383 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
W20240503 14:15:46.184175 28254 leader_election.cc:341] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:46.184338 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:46.184545 28057 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:46.185467 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:46.185561 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:46.185703 28319 raft_consensus.cc:2379] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7046cbd82d634976b53fe0cef5788f18 in current term 1: Already voted for candidate 74318b6270b64efaa79867167fc08bbb in this term.
I20240503 14:15:46.185789 27924 raft_consensus.cc:3037] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.186350 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:46.186538 28188 raft_consensus.cc:3037] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.186615 29544 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:46.186779 29544 ts_tablet_manager.cc:1367] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20240503 14:15:46.187597 29544 raft_consensus.cc:348] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.187871 29544 raft_consensus.cc:374] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.187949 29544 raft_consensus.cc:729] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.188184 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:46.188383 28451 raft_consensus.cc:3037] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.188266 29544 consensus_queue.cc:260] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.188643 27924 raft_consensus.cc:2454] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:46.189049 29544 ts_tablet_manager.cc:1398] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:15:46.191392 28451 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.192376 27598 leader_election.cc:302] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad
I20240503 14:15:46.193017 29379 raft_consensus.cc:2741] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 14:15:46.194272 29537 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:46.194451 29537 ts_tablet_manager.cc:1367] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.003s
I20240503 14:15:46.195281 29537 raft_consensus.cc:348] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.195490 28188 raft_consensus.cc:2454] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:46.195554 29537 raft_consensus.cc:374] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.195624 29537 raft_consensus.cc:729] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.199481 29537 consensus_queue.cc:260] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.201756 29537 ts_tablet_manager.cc:1398] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:15:46.202461 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:46.202704 28057 raft_consensus.cc:3037] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.205597 28057 raft_consensus.cc:2454] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:46.206250 28256 leader_election.cc:302] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 9231e715a4e342a197d6c5456ba6b399
I20240503 14:15:46.206456 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:46.206597 29492 raft_consensus.cc:2781] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:46.206661 27661 raft_consensus.cc:3037] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.206748 29492 raft_consensus.cc:686] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Becoming Leader. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Running, Role: LEADER
I20240503 14:15:46.207077 29492 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.210060 29494 leader_election.cc:288] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
W20240503 14:15:46.215691 28256 leader_election.cc:341] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:46.216293 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:46.216555 28188 raft_consensus.cc:3037] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.218972 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:46.219233 27792 raft_consensus.cc:3037] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.209223 27661 raft_consensus.cc:2454] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:46.219893 27513 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 74318b6270b64efaa79867167fc08bbb (127.10.145.2). New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:46.226518 29538 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:46.226744 29538 ts_tablet_manager.cc:1367] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.014s
I20240503 14:15:46.227301 28188 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.227684 29538 raft_consensus.cc:348] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.227890 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84acfe4f36f4193970fd2230229a41f" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:46.228009 29538 raft_consensus.cc:374] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.228132 28057 raft_consensus.cc:3037] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.228957 27792 raft_consensus.cc:2454] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb93936f2f14e628eecb05d7d1f4d91 in term 1.
I20240503 14:15:46.230518 28057 raft_consensus.cc:2454] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.231061 29538 raft_consensus.cc:729] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.230782 27514 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: term changed from 0 to 1, leader changed from <none> to acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6). New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:46.231896 29538 consensus_queue.cc:260] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.232585 29538 ts_tablet_manager.cc:1398] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240503 14:15:46.236924 29228 raft_consensus.cc:482] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:46.237123 29228 raft_consensus.cc:504] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:46.238533 29228 leader_election.cc:288] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:46.238616 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:46.238932 28188 raft_consensus.cc:2379] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9231e715a4e342a197d6c5456ba6b399 in current term 1: Already voted for candidate acb93936f2f14e628eecb05d7d1f4d91 in this term.
I20240503 14:15:46.238889 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:46.239715 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:46.239791 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:46.239871 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:46.240015 28057 raft_consensus.cc:2379] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9231e715a4e342a197d6c5456ba6b399 in current term 1: Already voted for candidate acb93936f2f14e628eecb05d7d1f4d91 in this term.
I20240503 14:15:46.240093 27924 raft_consensus.cc:2379] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9231e715a4e342a197d6c5456ba6b399 in current term 1: Already voted for candidate acb93936f2f14e628eecb05d7d1f4d91 in this term.
I20240503 14:15:46.240145 27661 raft_consensus.cc:2379] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9231e715a4e342a197d6c5456ba6b399 in current term 1: Already voted for candidate acb93936f2f14e628eecb05d7d1f4d91 in this term.
I20240503 14:15:46.240625 28387 leader_election.cc:302] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 9231e715a4e342a197d6c5456ba6b399; no voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422
I20240503 14:15:46.240733 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23af4dc19c412685927c70cbee7124" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:46.240936 29228 raft_consensus.cc:3037] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.241004 27792 raft_consensus.cc:2379] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9231e715a4e342a197d6c5456ba6b399 in current term 1: Already voted for candidate acb93936f2f14e628eecb05d7d1f4d91 in this term.
I20240503 14:15:46.243093 29228 raft_consensus.cc:2726] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:46.319305 29547 raft_consensus.cc:482] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:46.319548 29547 raft_consensus.cc:504] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.320537 29547 leader_election.cc:288] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:46.321082 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:46.321208 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:46.321323 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:46.321390 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:46.321509 27661 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 0.
I20240503 14:15:46.321553 28188 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 0.
I20240503 14:15:46.321498 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:46.321619 27924 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 0.
I20240503 14:15:46.321777 28451 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 0.
I20240503 14:15:46.322137 27729 leader_election.cc:302] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:46.321992 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:46.322326 28320 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 0.
I20240503 14:15:46.322453 28057 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 0.
I20240503 14:15:46.322471 29547 raft_consensus.cc:2781] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:46.322568 29547 raft_consensus.cc:482] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:46.322690 29547 raft_consensus.cc:3037] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.324805 29547 raft_consensus.cc:504] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.325937 29547 leader_election.cc:288] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 election: Requested vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:46.326061 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:46.326385 27661 raft_consensus.cc:3037] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.326339 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:46.326472 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:46.326534 28057 raft_consensus.cc:3037] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.326686 28320 raft_consensus.cc:3037] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.326665 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:46.326865 28188 raft_consensus.cc:3037] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.329331 28057 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.329339 27661 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.329859 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:46.329955 28188 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.329918 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b7e707cec54bc8bc5e649c3b37dfb5" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:46.330102 27924 raft_consensus.cc:3037] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.330140 28451 raft_consensus.cc:3037] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:46.330816 27729 leader_election.cc:302] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:46.331243 28320 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.331938 29547 raft_consensus.cc:2781] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:46.332440 29547 raft_consensus.cc:686] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Becoming Leader. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Running, Role: LEADER
I20240503 14:15:46.332882 28451 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.332976 27924 raft_consensus.cc:2454] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74318b6270b64efaa79867167fc08bbb in term 1.
I20240503 14:15:46.332870 29547 consensus_queue.cc:237] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:46.337227 27513 catalog_manager.cc:5484] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 74318b6270b64efaa79867167fc08bbb (127.10.145.2). New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:46.584028 29547 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.592401 29547 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.624473 29547 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.640075 29492 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.643527 29492 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.651448 29547 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.652593 28954 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.666761 29391 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.692341 29554 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.710923 29554 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.716989 29391 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.737982 29391 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:15:46.766551 29554 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.813551 29547 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.837584 29547 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.843799 29554 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.844687 29547 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:46.847712 29580 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:46.890795 29569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:46.891160 29569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:46.899067 29569 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 441.089 ms
time per row: 0.220545 ms
I20240503 14:15:47.497341 27513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51414:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:47.498354 27513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_5 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 14:15:47.508010 27773 tablet_service.cc:1467] Processing CreateTablet for tablet e895331729d3400384f20aa3895540de (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.508189 27641 tablet_service.cc:1467] Processing CreateTablet for tablet e895331729d3400384f20aa3895540de (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.508431 27905 tablet_service.cc:1467] Processing CreateTablet for tablet e895331729d3400384f20aa3895540de (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.508566 27773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e895331729d3400384f20aa3895540de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.508653 27641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e895331729d3400384f20aa3895540de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.508854 27905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e895331729d3400384f20aa3895540de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.509320 28430 tablet_service.cc:1467] Processing CreateTablet for tablet e895331729d3400384f20aa3895540de (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.509750 28430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e895331729d3400384f20aa3895540de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.514420 28298 tablet_service.cc:1467] Processing CreateTablet for tablet e895331729d3400384f20aa3895540de (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.514894 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e895331729d3400384f20aa3895540de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.518383 29669 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:47.518487 28167 tablet_service.cc:1467] Processing CreateTablet for tablet e895331729d3400384f20aa3895540de (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.518962 28167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e895331729d3400384f20aa3895540de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.522120 28036 tablet_service.cc:1467] Processing CreateTablet for tablet e895331729d3400384f20aa3895540de (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.522272 27772 tablet_service.cc:1467] Processing CreateTablet for tablet 56dad6fc0ba0440aa4660030b1bb9267 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.522624 28036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e895331729d3400384f20aa3895540de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.522686 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dad6fc0ba0440aa4660030b1bb9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.525840 28035 tablet_service.cc:1467] Processing CreateTablet for tablet 56dad6fc0ba0440aa4660030b1bb9267 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.525966 27774 tablet_service.cc:1467] Processing CreateTablet for tablet 48a05a802960425383984c379e238682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.526312 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dad6fc0ba0440aa4660030b1bb9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.526432 27774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a05a802960425383984c379e238682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.527199 29668 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:47.530884 29668 tablet_bootstrap.cc:654] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.530901 27906 tablet_service.cc:1467] Processing CreateTablet for tablet 56dad6fc0ba0440aa4660030b1bb9267 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.531404 27906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dad6fc0ba0440aa4660030b1bb9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.531817 28431 tablet_service.cc:1467] Processing CreateTablet for tablet 56dad6fc0ba0440aa4660030b1bb9267 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.532244 28431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dad6fc0ba0440aa4660030b1bb9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.533380 28430 tablet_service.cc:1467] Processing CreateTablet for tablet 48a05a802960425383984c379e238682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.533757 28430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a05a802960425383984c379e238682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.533697 27904 tablet_service.cc:1467] Processing CreateTablet for tablet 48a05a802960425383984c379e238682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.534109 27904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a05a802960425383984c379e238682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.539503 28299 tablet_service.cc:1467] Processing CreateTablet for tablet 56dad6fc0ba0440aa4660030b1bb9267 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.539819 28166 tablet_service.cc:1467] Processing CreateTablet for tablet 56dad6fc0ba0440aa4660030b1bb9267 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.539889 28168 tablet_service.cc:1467] Processing CreateTablet for tablet 48a05a802960425383984c379e238682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.540015 28299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dad6fc0ba0440aa4660030b1bb9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.540398 28166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dad6fc0ba0440aa4660030b1bb9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.542143 27642 tablet_service.cc:1467] Processing CreateTablet for tablet 56dad6fc0ba0440aa4660030b1bb9267 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.542354 29668 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:47.542477 28168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a05a802960425383984c379e238682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.542520 29668 ts_tablet_manager.cc:1367] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240503 14:15:47.542572 27642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dad6fc0ba0440aa4660030b1bb9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.543362 29668 raft_consensus.cc:348] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.544065 29668 raft_consensus.cc:374] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.544158 29668 raft_consensus.cc:729] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.544442 29668 consensus_queue.cc:260] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.545122 29668 ts_tablet_manager.cc:1398] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:15:47.545708 27640 tablet_service.cc:1467] Processing CreateTablet for tablet 48a05a802960425383984c379e238682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.546149 27640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a05a802960425383984c379e238682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.550359 28037 tablet_service.cc:1467] Processing CreateTablet for tablet 48a05a802960425383984c379e238682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.550844 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a05a802960425383984c379e238682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.555783 28300 tablet_service.cc:1467] Processing CreateTablet for tablet 48a05a802960425383984c379e238682 (DEFAULT_TABLE table=rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:47.556308 28300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a05a802960425383984c379e238682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:47.559597 29668 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:47.560312 29673 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:47.561441 29668 tablet_bootstrap.cc:654] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.562124 29673 tablet_bootstrap.cc:654] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.563519 29669 tablet_bootstrap.cc:654] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.564222 29668 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:47.564432 29668 ts_tablet_manager.cc:1367] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:15:47.565426 29674 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:47.565234 29668 raft_consensus.cc:348] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.565691 29668 raft_consensus.cc:374] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.565817 29668 raft_consensus.cc:729] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.566200 29668 consensus_queue.cc:260] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.566502 29673 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:47.566699 29673 ts_tablet_manager.cc:1367] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:15:47.567046 29668 ts_tablet_manager.cc:1398] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:47.567554 29673 raft_consensus.cc:348] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.567796 29673 raft_consensus.cc:374] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.567915 29673 raft_consensus.cc:729] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.568141 29673 consensus_queue.cc:260] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.568950 29673 ts_tablet_manager.cc:1398] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:47.569340 29673 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:47.571064 29673 tablet_bootstrap.cc:654] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.571120 29674 tablet_bootstrap.cc:654] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.574168 29554 raft_consensus.cc:482] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:47.574396 29554 raft_consensus.cc:504] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.575742 29554 leader_election.cc:288] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:47.576382 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
W20240503 14:15:47.577160 27728 leader_election.cc:341] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:15:47.577405 27729 leader_election.cc:341] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Not found: Tablet not found: e895331729d3400384f20aa3895540de
I20240503 14:15:47.576720 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
W20240503 14:15:47.577693 27730 leader_election.cc:341] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605): Not found: Tablet not found: e895331729d3400384f20aa3895540de
W20240503 14:15:47.578183 27730 leader_election.cc:341] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:47.578310 27730 leader_election.cc:302] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 74318b6270b64efaa79867167fc08bbb; no voters: 078f8eff84c94dde9705ad1330922a11, 9231e715a4e342a197d6c5456ba6b399, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422
I20240503 14:15:47.578581 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "74318b6270b64efaa79867167fc08bbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
W20240503 14:15:47.579303 27730 leader_election.cc:341] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119): Not found: Tablet not found: e895331729d3400384f20aa3895540de
W20240503 14:15:47.579723 27729 leader_election.cc:341] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:47.580842 29669 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:47.581959 29673 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:47.582203 29673 ts_tablet_manager.cc:1367] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240503 14:15:47.583089 29554 raft_consensus.cc:2726] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:47.583076 29673 raft_consensus.cc:348] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.583360 29669 ts_tablet_manager.cc:1367] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20240503 14:15:47.583539 29673 raft_consensus.cc:374] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.583694 29673 raft_consensus.cc:729] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.584022 29673 consensus_queue.cc:260] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.584823 29673 ts_tablet_manager.cc:1398] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:15:47.584646 29669 raft_consensus.cc:348] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.584913 29669 raft_consensus.cc:374] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.585184 29673 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:47.588685 29673 tablet_bootstrap.cc:654] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.590449 29669 raft_consensus.cc:729] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.590845 29674 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:47.590786 29669 consensus_queue.cc:260] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.591184 29674 ts_tablet_manager.cc:1367] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.005s
I20240503 14:15:47.591514 29669 ts_tablet_manager.cc:1398] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 14:15:47.591828 29669 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:47.592186 29674 raft_consensus.cc:348] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.592568 29674 raft_consensus.cc:374] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.592900 29674 raft_consensus.cc:729] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.593189 29674 consensus_queue.cc:260] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.598886 29669 tablet_bootstrap.cc:654] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.603439 29594 raft_consensus.cc:482] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:47.603652 29594 raft_consensus.cc:504] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.604957 29594 leader_election.cc:288] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
W20240503 14:15:47.605657 28123 leader_election.cc:341] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Not found: Tablet not found: 48a05a802960425383984c379e238682
I20240503 14:15:47.606200 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:47.606873 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:47.607107 27793 raft_consensus.cc:2454] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c91f285b8f4f4a27bcdd0ada303c1422 in term 0.
W20240503 14:15:47.607419 28124 leader_election.cc:341] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): Not found: Tablet not found: 48a05a802960425383984c379e238682
W20240503 14:15:47.607674 28123 leader_election.cc:341] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:47.608114 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
W20240503 14:15:47.608695 28122 leader_election.cc:341] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:47.608822 28122 leader_election.cc:302] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: c91f285b8f4f4a27bcdd0ada303c1422; no voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, acb93936f2f14e628eecb05d7d1f4d91
W20240503 14:15:47.609112 28124 leader_election.cc:341] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119): Not found: Tablet not found: 48a05a802960425383984c379e238682
I20240503 14:15:47.609596 29594 raft_consensus.cc:2726] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:47.611215 29674 ts_tablet_manager.cc:1398] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 14:15:47.621865 29674 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:47.623564 29668 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:47.624977 29674 tablet_bootstrap.cc:654] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.625243 29668 tablet_bootstrap.cc:654] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.642141 29668 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:47.642503 29668 ts_tablet_manager.cc:1367] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 14:15:47.645615 29668 raft_consensus.cc:348] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.646065 29668 raft_consensus.cc:374] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.646232 29668 raft_consensus.cc:729] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.646293 29672 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:47.646414 29682 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:47.646595 29668 consensus_queue.cc:260] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.648231 29668 ts_tablet_manager.cc:1398] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:15:47.650776 29674 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:47.650784 29682 tablet_bootstrap.cc:654] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.650988 29674 ts_tablet_manager.cc:1367] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.003s
I20240503 14:15:47.652863 29674 raft_consensus.cc:348] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.653577 29672 tablet_bootstrap.cc:654] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.654745 29670 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:47.655562 29669 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:47.655750 29674 raft_consensus.cc:374] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.655874 29674 raft_consensus.cc:729] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.656265 29674 consensus_queue.cc:260] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.656986 29674 ts_tablet_manager.cc:1398] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240503 14:15:47.657487 29674 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:47.658910 29669 ts_tablet_manager.cc:1367] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.000s
I20240503 14:15:47.660301 29674 tablet_bootstrap.cc:654] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.664312 29670 tablet_bootstrap.cc:654] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.665000 29669 raft_consensus.cc:348] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.666611 29669 raft_consensus.cc:374] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.666765 29669 raft_consensus.cc:729] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.670118 29672 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:47.671159 29672 ts_tablet_manager.cc:1367] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240503 14:15:47.672421 29669 consensus_queue.cc:260] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.673200 29669 ts_tablet_manager.cc:1398] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:15:47.674196 29672 raft_consensus.cc:348] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.674525 29672 raft_consensus.cc:374] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.674659 29672 raft_consensus.cc:729] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.674809 29682 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:47.675055 29682 ts_tablet_manager.cc:1367] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240503 14:15:47.674983 29672 consensus_queue.cc:260] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.675930 29672 ts_tablet_manager.cc:1398] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:15:47.678519 29669 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:47.677923 29682 raft_consensus.cc:348] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.680497 29669 tablet_bootstrap.cc:654] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.680660 29682 raft_consensus.cc:374] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.680833 29682 raft_consensus.cc:729] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.681164 29682 consensus_queue.cc:260] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.681999 29672 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:47.683748 29682 ts_tablet_manager.cc:1398] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 14:15:47.683780 29672 tablet_bootstrap.cc:654] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.684639 29673 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:47.685210 29673 ts_tablet_manager.cc:1367] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.100s user 0.028s sys 0.045s
I20240503 14:15:47.686128 29673 raft_consensus.cc:348] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.686471 29673 raft_consensus.cc:374] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.686597 29673 raft_consensus.cc:729] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.686975 29673 consensus_queue.cc:260] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.687717 29674 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:47.687887 29674 ts_tablet_manager.cc:1367] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.004s
I20240503 14:15:47.688359 29682 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:47.688717 29674 raft_consensus.cc:348] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.689004 29674 raft_consensus.cc:374] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.689107 29674 raft_consensus.cc:729] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.689405 29674 consensus_queue.cc:260] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.690013 29674 ts_tablet_manager.cc:1398] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:47.692111 29670 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:47.692358 29670 ts_tablet_manager.cc:1367] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240503 14:15:47.693392 29670 raft_consensus.cc:348] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.693768 29670 raft_consensus.cc:374] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.693892 29670 raft_consensus.cc:729] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.694192 29670 consensus_queue.cc:260] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.695813 29682 tablet_bootstrap.cc:654] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.696053 29673 ts_tablet_manager.cc:1398] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 14:15:47.701882 29670 ts_tablet_manager.cc:1398] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:15:47.702231 29670 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:47.703997 29670 tablet_bootstrap.cc:654] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.707072 29670 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:47.707376 29670 ts_tablet_manager.cc:1367] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:15:47.708076 29669 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:47.708305 29669 ts_tablet_manager.cc:1367] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240503 14:15:47.708276 29670 raft_consensus.cc:348] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.709059 29670 raft_consensus.cc:374] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.709102 29669 raft_consensus.cc:348] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.709437 29669 raft_consensus.cc:374] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.709340 29670 raft_consensus.cc:729] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.709555 29669 raft_consensus.cc:729] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.709857 29669 consensus_queue.cc:260] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.710521 29669 ts_tablet_manager.cc:1398] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:47.709951 29670 consensus_queue.cc:260] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.712337 29670 ts_tablet_manager.cc:1398] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:15:47.713397 29670 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:47.714682 29682 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:47.714905 29682 ts_tablet_manager.cc:1367] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.004s
I20240503 14:15:47.715682 29682 raft_consensus.cc:348] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.715978 29682 raft_consensus.cc:374] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.716042 29682 raft_consensus.cc:729] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.716297 29682 consensus_queue.cc:260] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.716877 29682 ts_tablet_manager.cc:1398] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:47.717202 29682 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:47.718765 29228 raft_consensus.cc:482] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:47.718966 29228 raft_consensus.cc:504] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.720198 29228 leader_election.cc:288] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:15:47.720830 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:47.721064 28057 raft_consensus.cc:2454] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:47.721156 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:47.721694 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:47.721788 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:47.721913 28187 raft_consensus.cc:2454] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
W20240503 14:15:47.721980 28386 leader_election.cc:341] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:47.721943 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:47.722011 27793 raft_consensus.cc:2454] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:47.722169 27926 raft_consensus.cc:2454] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:47.722687 28387 leader_election.cc:302] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:47.723198 29228 raft_consensus.cc:2781] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:47.723294 29228 raft_consensus.cc:482] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:47.723385 29228 raft_consensus.cc:3037] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.725020 29682 tablet_bootstrap.cc:654] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.726115 29228 raft_consensus.cc:504] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.722594 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
W20240503 14:15:47.727212 28386 leader_election.cc:341] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:47.727413 29672 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:47.727603 29672 ts_tablet_manager.cc:1367] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.013s
I20240503 14:15:47.727758 29682 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:47.727921 29682 ts_tablet_manager.cc:1367] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240503 14:15:47.728111 29670 tablet_bootstrap.cc:654] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.728405 29672 raft_consensus.cc:348] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.728735 29672 raft_consensus.cc:374] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.728636 29682 raft_consensus.cc:348] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.728829 29672 raft_consensus.cc:729] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.728924 29682 raft_consensus.cc:374] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.729044 29682 raft_consensus.cc:729] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.729106 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:47.729158 29672 consensus_queue.cc:260] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.729360 29682 consensus_queue.cc:260] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.730090 29682 ts_tablet_manager.cc:1398] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:47.730098 29672 ts_tablet_manager.cc:1398] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:47.730422 29672 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
W20240503 14:15:47.731012 28386 leader_election.cc:341] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:47.731995 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:47.732244 27793 raft_consensus.cc:3037] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.735410 27793 raft_consensus.cc:2454] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:47.736742 29672 tablet_bootstrap.cc:654] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.738152 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:47.738394 27926 raft_consensus.cc:3037] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.742118 29325 raft_consensus.cc:482] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:47.742327 29325 raft_consensus.cc:504] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.743722 29325 leader_election.cc:288] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:47.744700 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:47.744985 28187 raft_consensus.cc:2454] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:47.745069 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:47.745375 28057 raft_consensus.cc:2454] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:47.747349 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
W20240503 14:15:47.748301 27861 leader_election.cc:341] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:47.752280 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:47.752647 27661 raft_consensus.cc:2454] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:47.753500 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:47.753499 27861 leader_election.cc:302] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:47.753803 28451 raft_consensus.cc:2454] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:47.753947 29325 raft_consensus.cc:2781] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:47.754060 29325 raft_consensus.cc:482] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:47.754148 29325 raft_consensus.cc:3037] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.755020 27926 raft_consensus.cc:2454] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:47.755911 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:47.756327 27793 raft_consensus.cc:2454] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:47.758180 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:47.758421 28187 raft_consensus.cc:3037] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.758423 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:47.758074 29325 raft_consensus.cc:504] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.758652 28320 raft_consensus.cc:3037] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.759912 29325 leader_election.cc:288] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 election: Requested vote from peers 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:47.761314 28187 raft_consensus.cc:2454] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:47.764607 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:47.764919 27792 raft_consensus.cc:3037] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.768184 27792 raft_consensus.cc:2454] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 1.
I20240503 14:15:47.770612 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:47.770893 28057 raft_consensus.cc:3037] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.774274 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:47.774580 28187 raft_consensus.cc:3037] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.777956 28387 leader_election.cc:302] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: 7046cbd82d634976b53fe0cef5788f18
I20240503 14:15:47.780133 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:47.780424 29697 raft_consensus.cc:2781] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:47.780571 29697 raft_consensus.cc:686] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Becoming Leader. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Running, Role: LEADER
I20240503 14:15:47.780813 29325 raft_consensus.cc:482] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:47.781004 29325 raft_consensus.cc:504] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.781635 29697 consensus_queue.cc:237] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.782250 29325 leader_election.cc:288] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859)
I20240503 14:15:47.782939 29661 raft_consensus.cc:482] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:47.783126 29661 raft_consensus.cc:504] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.784524 29661 leader_election.cc:288] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:47.785517 28186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:47.785751 28186 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:47.785835 28055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:47.786098 28055 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:47.786599 27794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:47.786847 27794 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:47.787180 28055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:47.787393 28055 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:47.787444 28186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:47.787650 28186 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
W20240503 14:15:47.787947 27861 leader_election.cc:341] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:47.788281 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:47.788532 28319 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:47.788486 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:47.788703 27662 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:47.788964 27597 leader_election.cc:302] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:47.789345 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:47.789547 28319 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 0.
I20240503 14:15:47.789741 27862 leader_election.cc:302] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:47.790004 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:47.790216 27661 raft_consensus.cc:3037] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.793524 29494 raft_consensus.cc:2781] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:47.793650 29494 raft_consensus.cc:482] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:47.793731 29494 raft_consensus.cc:3037] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.794322 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:47.794556 28451 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:47.799260 29325 raft_consensus.cc:2781] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:47.799448 29325 raft_consensus.cc:482] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:47.799535 29325 raft_consensus.cc:3037] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.807945 28057 raft_consensus.cc:2454] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 1.
I20240503 14:15:47.808512 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:47.809168 28320 raft_consensus.cc:2454] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:47.809877 28187 raft_consensus.cc:2454] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 1.
I20240503 14:15:47.810861 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:47.811237 27792 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 0.
I20240503 14:15:47.813867 27862 leader_election.cc:302] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 74318b6270b64efaa79867167fc08bbb, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422; no voters: acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:47.821885 29325 raft_consensus.cc:504] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.836896 29494 raft_consensus.cc:504] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.837512 27661 raft_consensus.cc:2454] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 1.
I20240503 14:15:47.840166 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:47.840556 28057 raft_consensus.cc:3037] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.840888 28186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:47.841117 28186 raft_consensus.cc:3037] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.843902 28186 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:47.844383 28057 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:47.846408 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:47.846701 27793 raft_consensus.cc:3037] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.846782 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:47.847013 28450 raft_consensus.cc:3037] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.847123 28320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:47.847308 28320 raft_consensus.cc:3037] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.849969 27793 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:47.839169 29494 leader_election.cc:288] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Requested vote from peers acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:47.850795 27598 leader_election.cc:302] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:47.851168 29658 raft_consensus.cc:2781] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:47.851298 29658 raft_consensus.cc:686] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Becoming Leader. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Running, Role: LEADER
I20240503 14:15:47.851665 29658 consensus_queue.cc:237] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:47.852926 28450 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:47.859822 29228 leader_election.cc:288] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 election: Requested vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:15:47.860366 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a05a802960425383984c379e238682" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:47.860618 28057 raft_consensus.cc:3037] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.860100 27518 catalog_manager.cc:5484] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1). New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:47.846629 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "7046cbd82d634976b53fe0cef5788f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:47.864202 28057 raft_consensus.cc:2454] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:47.868602 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:47.869702 27926 raft_consensus.cc:2379] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7046cbd82d634976b53fe0cef5788f18 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:47.874658 28320 raft_consensus.cc:2454] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7046cbd82d634976b53fe0cef5788f18 in term 1.
I20240503 14:15:47.876987 29325 leader_election.cc:288] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 election: Requested vote from peers acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859)
I20240503 14:15:47.878419 29325 raft_consensus.cc:2781] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:47.878671 29325 raft_consensus.cc:686] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Becoming Leader. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Running, Role: LEADER
I20240503 14:15:47.877319 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:47.879150 29325 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.896150 28450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:47.896478 28450 raft_consensus.cc:3037] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:15:47.896759 27862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.145.5:35605, user_credentials={real_user=slave}} blocked reactor thread for 86374us
I20240503 14:15:47.897945 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:47.898219 28187 raft_consensus.cc:2379] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a242f99f54b4494893064cfe3030b4ad in current term 1: Already voted for candidate 7046cbd82d634976b53fe0cef5788f18 in this term.
I20240503 14:15:47.899425 28450 raft_consensus.cc:2454] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a242f99f54b4494893064cfe3030b4ad in term 1.
I20240503 14:15:47.899906 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:47.900209 28057 raft_consensus.cc:2379] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a242f99f54b4494893064cfe3030b4ad in current term 1: Already voted for candidate 7046cbd82d634976b53fe0cef5788f18 in this term.
I20240503 14:15:47.896188 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:47.901438 28451 raft_consensus.cc:2379] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a242f99f54b4494893064cfe3030b4ad in current term 1: Already voted for candidate 7046cbd82d634976b53fe0cef5788f18 in this term.
I20240503 14:15:47.901471 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:47.901722 27793 raft_consensus.cc:2379] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a242f99f54b4494893064cfe3030b4ad in current term 1: Already voted for candidate 7046cbd82d634976b53fe0cef5788f18 in this term.
I20240503 14:15:47.902565 27518 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad reported cstate change: term changed from 0 to 1, leader changed from <none> to a242f99f54b4494893064cfe3030b4ad (127.10.145.3). New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:47.906788 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56dad6fc0ba0440aa4660030b1bb9267" candidate_uuid: "a242f99f54b4494893064cfe3030b4ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:47.907099 28451 raft_consensus.cc:2379] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a242f99f54b4494893064cfe3030b4ad in current term 1: Already voted for candidate 7046cbd82d634976b53fe0cef5788f18 in this term.
I20240503 14:15:47.908020 27862 leader_election.cc:302] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: a242f99f54b4494893064cfe3030b4ad; no voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:15:47.908713 29701 raft_consensus.cc:2726] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:15:47.914278 29672 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:47.914482 29672 ts_tablet_manager.cc:1367] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.184s user 0.039s sys 0.091s
I20240503 14:15:47.914637 27518 catalog_manager.cc:5484] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7). New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:47.915340 29672 raft_consensus.cc:348] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.915607 29672 raft_consensus.cc:374] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.915709 29672 raft_consensus.cc:729] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.915975 29670 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:47.916132 29670 ts_tablet_manager.cc:1367] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.203s user 0.056s sys 0.061s
I20240503 14:15:47.915992 29672 consensus_queue.cc:260] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.916705 29672 ts_tablet_manager.cc:1398] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:15:47.917011 29670 raft_consensus.cc:348] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.917266 29670 raft_consensus.cc:374] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.917377 29670 raft_consensus.cc:729] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.917672 29670 consensus_queue.cc:260] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:47.918370 29670 ts_tablet_manager.cc:1398] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:15:48.171520 29696 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.174866 29492 raft_consensus.cc:482] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:48.175127 29492 raft_consensus.cc:504] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:48.176337 29492 leader_election.cc:288] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:48.178826 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:48.179260 28187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:48.179562 28187 raft_consensus.cc:2379] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acb93936f2f14e628eecb05d7d1f4d91 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:48.179967 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:48.180258 28057 raft_consensus.cc:2379] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acb93936f2f14e628eecb05d7d1f4d91 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:48.181138 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:48.181867 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:48.182168 27661 raft_consensus.cc:2379] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acb93936f2f14e628eecb05d7d1f4d91 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:48.182600 28451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e895331729d3400384f20aa3895540de" candidate_uuid: "acb93936f2f14e628eecb05d7d1f4d91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:48.182885 28451 raft_consensus.cc:2379] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acb93936f2f14e628eecb05d7d1f4d91 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:48.183061 28255 leader_election.cc:302] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: acb93936f2f14e628eecb05d7d1f4d91; no voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, a242f99f54b4494893064cfe3030b4ad, c91f285b8f4f4a27bcdd0ada303c1422
I20240503 14:15:48.183612 27792 raft_consensus.cc:2379] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate acb93936f2f14e628eecb05d7d1f4d91 in current term 1: Already voted for candidate a242f99f54b4494893064cfe3030b4ad in this term.
I20240503 14:15:48.184105 29492 raft_consensus.cc:3037] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.186893 29492 raft_consensus.cc:2726] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:48.280392 29658 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.318199 29696 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.321313 29661 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.335312 29701 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.352625 29701 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.356864 29699 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.363287 27661 raft_consensus.cc:3037] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.365590 29699 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.370463 29699 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.378185 29661 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.394057 29325 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.412631 29325 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.416586 29699 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.417438 29727 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.444257 29325 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.450177 29727 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:48.474758 27596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:43675, user_credentials={real_user=slave}} blocked reactor thread for 67707.2us
I20240503 14:15:48.486249 29661 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.486783 29701 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:48.573954 29704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:48.574390 29704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:48.584362 29704 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 445.135 ms
time per row: 0.222567 ms
I20240503 14:15:49.276554 27513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51414:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 14:15:49.277514 27513 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_6 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 14:15:49.287650 27641 tablet_service.cc:1467] Processing CreateTablet for tablet 9f00380931064f71bc78626af72488f3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.287840 27772 tablet_service.cc:1467] Processing CreateTablet for tablet 9f00380931064f71bc78626af72488f3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.288152 27641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f00380931064f71bc78626af72488f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.288311 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f00380931064f71bc78626af72488f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.288282 27904 tablet_service.cc:1467] Processing CreateTablet for tablet 9f00380931064f71bc78626af72488f3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.288697 27904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f00380931064f71bc78626af72488f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.290356 28167 tablet_service.cc:1467] Processing CreateTablet for tablet 9f00380931064f71bc78626af72488f3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.290794 28167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f00380931064f71bc78626af72488f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.293543 28035 tablet_service.cc:1467] Processing CreateTablet for tablet 9f00380931064f71bc78626af72488f3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.294008 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f00380931064f71bc78626af72488f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.303449 29818 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:49.304108 29819 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:49.304749 28299 tablet_service.cc:1467] Processing CreateTablet for tablet 9f00380931064f71bc78626af72488f3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.305241 28299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f00380931064f71bc78626af72488f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.305948 29819 tablet_bootstrap.cc:654] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.306052 28431 tablet_service.cc:1467] Processing CreateTablet for tablet 9f00380931064f71bc78626af72488f3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.306556 28431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f00380931064f71bc78626af72488f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.307991 29820 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:49.308918 29818 tablet_bootstrap.cc:654] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.311184 29819 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:49.311353 29819 ts_tablet_manager.cc:1367] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:15:49.312194 29819 raft_consensus.cc:348] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.312484 29819 raft_consensus.cc:374] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.312570 29819 raft_consensus.cc:729] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.312819 29819 consensus_queue.cc:260] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.313527 29819 ts_tablet_manager.cc:1398] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:49.314762 29820 tablet_bootstrap.cc:654] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.317692 27641 tablet_service.cc:1467] Processing CreateTablet for tablet 44df52bbf3534802bf7217fb5cdb4326 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.318218 27641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44df52bbf3534802bf7217fb5cdb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.323572 28300 tablet_service.cc:1467] Processing CreateTablet for tablet 44df52bbf3534802bf7217fb5cdb4326 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.324179 28300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44df52bbf3534802bf7217fb5cdb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.330180 27904 tablet_service.cc:1467] Processing CreateTablet for tablet 44df52bbf3534802bf7217fb5cdb4326 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.330405 27772 tablet_service.cc:1467] Processing CreateTablet for tablet 44df52bbf3534802bf7217fb5cdb4326 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.330669 27904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44df52bbf3534802bf7217fb5cdb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.330804 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44df52bbf3534802bf7217fb5cdb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.337838 29819 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:49.339788 29819 tablet_bootstrap.cc:654] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.341087 28168 tablet_service.cc:1467] Processing CreateTablet for tablet 44df52bbf3534802bf7217fb5cdb4326 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.341673 28168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44df52bbf3534802bf7217fb5cdb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.348073 29825 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:49.351266 29827 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:49.354943 28430 tablet_service.cc:1467] Processing CreateTablet for tablet 44df52bbf3534802bf7217fb5cdb4326 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.355489 28430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44df52bbf3534802bf7217fb5cdb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.355578 27642 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9c369355ad47f4843da19dc3ae3cac (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.355980 27642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9c369355ad47f4843da19dc3ae3cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.357735 28300 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9c369355ad47f4843da19dc3ae3cac (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.358157 28300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9c369355ad47f4843da19dc3ae3cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.359305 28168 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9c369355ad47f4843da19dc3ae3cac (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.359556 29828 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:49.359800 28168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9c369355ad47f4843da19dc3ae3cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.360047 27772 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9c369355ad47f4843da19dc3ae3cac (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.360919 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9c369355ad47f4843da19dc3ae3cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.362140 29827 tablet_bootstrap.cc:654] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.364801 27904 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9c369355ad47f4843da19dc3ae3cac (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.365243 27904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9c369355ad47f4843da19dc3ae3cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.366746 29828 tablet_bootstrap.cc:654] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.368613 28037 tablet_service.cc:1467] Processing CreateTablet for tablet 44df52bbf3534802bf7217fb5cdb4326 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.369082 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44df52bbf3534802bf7217fb5cdb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.369920 28430 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9c369355ad47f4843da19dc3ae3cac (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.370393 28430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9c369355ad47f4843da19dc3ae3cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.371492 29819 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:49.371577 29825 tablet_bootstrap.cc:654] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.371675 29819 ts_tablet_manager.cc:1367] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.000s
I20240503 14:15:49.374032 29819 raft_consensus.cc:348] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.374441 29819 raft_consensus.cc:374] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.374567 29819 raft_consensus.cc:729] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.374701 28035 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9c369355ad47f4843da19dc3ae3cac (DEFAULT_TABLE table=rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:49.374925 29819 consensus_queue.cc:260] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.375350 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9c369355ad47f4843da19dc3ae3cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:49.375748 29819 ts_tablet_manager.cc:1398] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:15:49.382409 29819 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb: Bootstrap starting.
I20240503 14:15:49.384469 29819 tablet_bootstrap.cc:654] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.388448 29819 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb: No bootstrap required, opened a new log
I20240503 14:15:49.388640 29819 ts_tablet_manager.cc:1367] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:15:49.389555 29819 raft_consensus.cc:348] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.389884 29819 raft_consensus.cc:374] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.392287 29819 raft_consensus.cc:729] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74318b6270b64efaa79867167fc08bbb, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.392603 29819 consensus_queue.cc:260] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.393553 29819 ts_tablet_manager.cc:1398] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:15:49.394619 29818 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:49.394872 29818 ts_tablet_manager.cc:1367] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.092s user 0.015s sys 0.014s
I20240503 14:15:49.395835 29818 raft_consensus.cc:348] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.396265 29818 raft_consensus.cc:374] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.396395 29818 raft_consensus.cc:729] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.396749 29818 consensus_queue.cc:260] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.398694 29818 ts_tablet_manager.cc:1398] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:15:49.399015 29828 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:49.399057 29818 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:49.400107 29825 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:49.400449 29828 ts_tablet_manager.cc:1367] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20240503 14:15:49.400743 29818 tablet_bootstrap.cc:654] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.400848 29825 ts_tablet_manager.cc:1367] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.000s
I20240503 14:15:49.401859 29828 raft_consensus.cc:348] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.402202 29828 raft_consensus.cc:374] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.402287 29828 raft_consensus.cc:729] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.402544 29828 consensus_queue.cc:260] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.403843 29825 raft_consensus.cc:348] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.407128 29825 raft_consensus.cc:374] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.407683 29825 raft_consensus.cc:729] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.408180 29823 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:49.408596 29825 consensus_queue.cc:260] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.409536 29828 ts_tablet_manager.cc:1398] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:15:49.409950 29828 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:49.411639 29828 tablet_bootstrap.cc:654] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.412272 29823 tablet_bootstrap.cc:654] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.413695 29825 ts_tablet_manager.cc:1398] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:15:49.415510 29823 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:49.416602 29825 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:49.417531 29823 ts_tablet_manager.cc:1367] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20240503 14:15:49.418567 29825 tablet_bootstrap.cc:654] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.419148 29820 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:49.419332 29820 ts_tablet_manager.cc:1367] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.016s
I20240503 14:15:49.420114 29820 raft_consensus.cc:348] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.420400 29820 raft_consensus.cc:374] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.420500 29820 raft_consensus.cc:729] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.420545 29823 raft_consensus.cc:348] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.420782 29820 consensus_queue.cc:260] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.420994 29823 raft_consensus.cc:374] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.421185 29823 raft_consensus.cc:729] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.421584 29820 ts_tablet_manager.cc:1398] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:15:49.421525 29823 consensus_queue.cc:260] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.421926 29820 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:49.423862 29823 ts_tablet_manager.cc:1398] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:15:49.424868 29823 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:49.427134 29827 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:49.427734 29820 tablet_bootstrap.cc:654] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.427994 29827 ts_tablet_manager.cc:1367] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.004s
I20240503 14:15:49.428968 29823 tablet_bootstrap.cc:654] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.430405 29818 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:49.430609 29818 ts_tablet_manager.cc:1367] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.003s
I20240503 14:15:49.430658 29827 raft_consensus.cc:348] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.431447 29818 raft_consensus.cc:348] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.431780 29818 raft_consensus.cc:374] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.431855 29818 raft_consensus.cc:729] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.432051 29827 raft_consensus.cc:374] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.432132 29818 consensus_queue.cc:260] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.432431 29827 raft_consensus.cc:729] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.432772 29818 ts_tablet_manager.cc:1398] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:15:49.432747 29827 consensus_queue.cc:260] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.433585 29827 ts_tablet_manager.cc:1398] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:15:49.434441 29818 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad: Bootstrap starting.
I20240503 14:15:49.437157 29818 tablet_bootstrap.cc:654] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.437273 29827 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:49.439204 29827 tablet_bootstrap.cc:654] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.441403 29828 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:49.441664 29828 ts_tablet_manager.cc:1367] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.006s
I20240503 14:15:49.442654 29823 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:49.442554 29828 raft_consensus.cc:348] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.442889 29823 ts_tablet_manager.cc:1367] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240503 14:15:49.442997 29828 raft_consensus.cc:374] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.443102 29828 raft_consensus.cc:729] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.443416 29828 consensus_queue.cc:260] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.444442 29828 ts_tablet_manager.cc:1398] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:49.444883 29828 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399: Bootstrap starting.
I20240503 14:15:49.444911 29825 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:49.445082 29825 ts_tablet_manager.cc:1367] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.008s
I20240503 14:15:49.445858 29825 raft_consensus.cc:348] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.446173 29825 raft_consensus.cc:374] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.446266 29825 raft_consensus.cc:729] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.446555 29825 consensus_queue.cc:260] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.447094 29828 tablet_bootstrap.cc:654] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.447996 29825 ts_tablet_manager.cc:1398] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:49.449056 29818 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad: No bootstrap required, opened a new log
I20240503 14:15:49.443804 29823 raft_consensus.cc:348] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.450073 29823 raft_consensus.cc:374] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.450186 29823 raft_consensus.cc:729] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.450500 29823 consensus_queue.cc:260] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.451712 29818 ts_tablet_manager.cc:1367] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240503 14:15:49.451993 29823 ts_tablet_manager.cc:1398] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20240503 14:15:49.452316 29823 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:15:49.452592 29818 raft_consensus.cc:348] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.452904 29818 raft_consensus.cc:374] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.452991 29818 raft_consensus.cc:729] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a242f99f54b4494893064cfe3030b4ad, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.453361 29818 consensus_queue.cc:260] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.455050 29818 ts_tablet_manager.cc:1398] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:49.455170 29823 tablet_bootstrap.cc:654] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.459112 29825 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:15:49.459517 29823 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91: No bootstrap required, opened a new log
I20240503 14:15:49.459702 29823 ts_tablet_manager.cc:1367] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240503 14:15:49.461238 29825 tablet_bootstrap.cc:654] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.461555 29820 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:49.461763 29820 ts_tablet_manager.cc:1367] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.003s
I20240503 14:15:49.462190 29823 raft_consensus.cc:348] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.462492 29823 raft_consensus.cc:374] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.462599 29823 raft_consensus.cc:729] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.462545 29820 raft_consensus.cc:348] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.462826 29820 raft_consensus.cc:374] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.462909 29820 raft_consensus.cc:729] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.463215 29820 consensus_queue.cc:260] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.463443 29827 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:49.463614 29827 ts_tablet_manager.cc:1367] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240503 14:15:49.463888 29820 ts_tablet_manager.cc:1398] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:49.464239 29820 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:15:49.464805 29828 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399: No bootstrap required, opened a new log
I20240503 14:15:49.465057 29828 ts_tablet_manager.cc:1367] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20240503 14:15:49.466099 29820 tablet_bootstrap.cc:654] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.466013 29828 raft_consensus.cc:348] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.464398 29827 raft_consensus.cc:348] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.466990 29825 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11: No bootstrap required, opened a new log
I20240503 14:15:49.467075 29827 raft_consensus.cc:374] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.467144 29825 ts_tablet_manager.cc:1367] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240503 14:15:49.467165 29827 raft_consensus.cc:729] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.467460 29827 consensus_queue.cc:260] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.466445 29828 raft_consensus.cc:374] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.468173 29828 raft_consensus.cc:729] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.468552 29828 consensus_queue.cc:260] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.470050 29828 ts_tablet_manager.cc:1398] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:15:49.471141 29825 raft_consensus.cc:348] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.471434 29825 raft_consensus.cc:374] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.471530 29825 raft_consensus.cc:729] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.471772 29825 consensus_queue.cc:260] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.472641 29827 ts_tablet_manager.cc:1398] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240503 14:15:49.473188 29827 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422: Bootstrap starting.
I20240503 14:15:49.475011 29825 ts_tablet_manager.cc:1398] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 14:15:49.477075 29827 tablet_bootstrap.cc:654] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:49.478040 29820 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18: No bootstrap required, opened a new log
I20240503 14:15:49.478278 29820 ts_tablet_manager.cc:1367] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240503 14:15:49.478327 29823 consensus_queue.cc:260] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.479386 29820 raft_consensus.cc:348] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.479723 29823 ts_tablet_manager.cc:1398] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.020s user 0.001s sys 0.002s
I20240503 14:15:49.479722 29820 raft_consensus.cc:374] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.479826 29820 raft_consensus.cc:729] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.480078 29820 consensus_queue.cc:260] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.480599 29820 ts_tablet_manager.cc:1398] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:49.483809 29730 raft_consensus.cc:482] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:49.483995 29730 raft_consensus.cc:504] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.484344 29827 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422: No bootstrap required, opened a new log
I20240503 14:15:49.484514 29827 ts_tablet_manager.cc:1367] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 14:15:49.485308 29730 leader_election.cc:288] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:49.485374 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:49.485591 27662 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:49.485733 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:49.485958 27793 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:49.486387 28449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399" is_pre_election: true
I20240503 14:15:49.486613 28449 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:49.486788 28186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:49.487028 28186 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:49.486972 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:49.487215 27926 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:49.487270 29827 raft_consensus.cc:348] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.487529 29827 raft_consensus.cc:374] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:49.487494 27993 leader_election.cc:302] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:49.487639 29827 raft_consensus.cc:729] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91f285b8f4f4a27bcdd0ada303c1422, State: Initialized, Role: FOLLOWER
I20240503 14:15:49.487779 29730 raft_consensus.cc:2781] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:49.487885 29730 raft_consensus.cc:482] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:49.487969 29730 raft_consensus.cc:3037] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.487948 29827 consensus_queue.cc:260] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.488663 29827 ts_tablet_manager.cc:1398] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:15:49.488610 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:49.488874 28319 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 0.
I20240503 14:15:49.491072 29730 raft_consensus.cc:504] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.492244 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:49.492348 29730 leader_election.cc:288] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 election: Requested vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:49.492488 27662 raft_consensus.cc:3037] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.492449 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:49.492641 27926 raft_consensus.cc:3037] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.492764 28186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:49.492992 28186 raft_consensus.cc:3037] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.493695 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:49.493908 27793 raft_consensus.cc:3037] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.495419 27662 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:49.495592 28186 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:49.496120 28449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9231e715a4e342a197d6c5456ba6b399"
I20240503 14:15:49.496326 28449 raft_consensus.cc:3037] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.498271 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f00380931064f71bc78626af72488f3" candidate_uuid: "078f8eff84c94dde9705ad1330922a11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:49.498448 28319 raft_consensus.cc:3037] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.498797 28449 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:49.499181 27991 leader_election.cc:302] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:49.499476 29730 raft_consensus.cc:2781] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:49.499641 29730 raft_consensus.cc:686] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Becoming Leader. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Running, Role: LEADER
I20240503 14:15:49.500162 29730 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.500536 28319 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:49.502552 27926 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:49.503986 27793 raft_consensus.cc:2454] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078f8eff84c94dde9705ad1330922a11 in term 1.
I20240503 14:15:49.504264 27513 catalog_manager.cc:5484] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 078f8eff84c94dde9705ad1330922a11 (127.10.145.4). New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:49.529033 29811 raft_consensus.cc:482] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:49.529261 29811 raft_consensus.cc:504] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.530434 29811 leader_election.cc:288] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:15:49.531080 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:49.531343 27662 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.531477 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:49.531759 28057 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.531728 28186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:49.531980 28186 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.532388 28387 leader_election.cc:302] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:49.532685 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:49.532809 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:49.532958 29811 raft_consensus.cc:2781] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:49.532997 28319 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.533074 29811 raft_consensus.cc:482] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:49.533051 27793 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.533151 29811 raft_consensus.cc:3037] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.532450 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:49.534839 27926 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.535447 29811 raft_consensus.cc:504] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.536685 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:49.536758 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:49.536945 27662 raft_consensus.cc:3037] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.536926 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:49.536929 28186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:49.537144 28186 raft_consensus.cc:3037] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.537164 28057 raft_consensus.cc:3037] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.537420 27926 raft_consensus.cc:3037] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.539116 29811 leader_election.cc:288] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 election: Requested vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:15:49.539510 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:49.539757 28057 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.539765 27662 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.539927 28319 raft_consensus.cc:3037] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.540784 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9c369355ad47f4843da19dc3ae3cac" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:49.541111 27793 raft_consensus.cc:3037] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.541103 28186 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.541342 27926 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.541687 28387 leader_election.cc:302] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 9231e715a4e342a197d6c5456ba6b399, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:49.542062 29811 raft_consensus.cc:2781] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:49.542402 29811 raft_consensus.cc:686] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Becoming Leader. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Running, Role: LEADER
I20240503 14:15:49.543046 29811 consensus_queue.cc:237] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.543181 28319 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.544288 27793 raft_consensus.cc:2454] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.547052 27513 catalog_manager.cc:5484] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7). New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:49.570380 29811 raft_consensus.cc:482] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:49.570628 29811 raft_consensus.cc:504] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.572078 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18" is_pre_election: true
I20240503 14:15:49.572402 27662 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.572387 27924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad" is_pre_election: true
I20240503 14:15:49.572422 28186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" is_pre_election: true
I20240503 14:15:49.572643 28186 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.572647 27924 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.572081 27793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb" is_pre_election: true
I20240503 14:15:49.573143 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91" is_pre_election: true
I20240503 14:15:49.573513 28319 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.573508 27793 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.574110 28386 leader_election.cc:302] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 9231e715a4e342a197d6c5456ba6b399, a242f99f54b4494893064cfe3030b4ad, acb93936f2f14e628eecb05d7d1f4d91, c91f285b8f4f4a27bcdd0ada303c1422; no voters:
I20240503 14:15:49.574126 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11" is_pre_election: true
I20240503 14:15:49.574370 28057 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 0.
I20240503 14:15:49.574589 29811 leader_election.cc:288] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:15:49.574649 29696 raft_consensus.cc:2781] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:49.574753 29696 raft_consensus.cc:482] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:49.574837 29696 raft_consensus.cc:3037] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.576948 29696 raft_consensus.cc:504] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.578001 29696 leader_election.cc:288] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 election: Requested vote from peers 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261), 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859), a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119), c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605), acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073), 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:15:49.578159 27792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74318b6270b64efaa79867167fc08bbb"
I20240503 14:15:49.578234 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7046cbd82d634976b53fe0cef5788f18"
I20240503 14:15:49.578399 27792 raft_consensus.cc:3037] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.578500 27662 raft_consensus.cc:3037] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.578637 28186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c91f285b8f4f4a27bcdd0ada303c1422"
I20240503 14:15:49.578703 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078f8eff84c94dde9705ad1330922a11"
I20240503 14:15:49.578881 28186 raft_consensus.cc:3037] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.578907 28057 raft_consensus.cc:3037] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.581271 27792 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.581372 27662 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.581485 27926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a242f99f54b4494893064cfe3030b4ad"
I20240503 14:15:49.581667 28057 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.581727 27926 raft_consensus.cc:3037] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.582994 28386 leader_election.cc:302] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 078f8eff84c94dde9705ad1330922a11, 7046cbd82d634976b53fe0cef5788f18, 74318b6270b64efaa79867167fc08bbb, 9231e715a4e342a197d6c5456ba6b399; no voters:
I20240503 14:15:49.583375 29696 raft_consensus.cc:2781] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:49.583398 28319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44df52bbf3534802bf7217fb5cdb4326" candidate_uuid: "9231e715a4e342a197d6c5456ba6b399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb93936f2f14e628eecb05d7d1f4d91"
I20240503 14:15:49.583570 29696 raft_consensus.cc:686] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Becoming Leader. State: Replica: 9231e715a4e342a197d6c5456ba6b399, State: Running, Role: LEADER
I20240503 14:15:49.583683 28319 raft_consensus.cc:3037] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:49.583921 29696 consensus_queue.cc:237] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:49.584321 28186 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.585217 27926 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.587306 28319 raft_consensus.cc:2454] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231e715a4e342a197d6c5456ba6b399 in term 1.
I20240503 14:15:49.588544 27513 catalog_manager.cc:5484] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7). New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:49.893563 29730 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:49.926517 29730 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:49.948887 29696 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:49.988329 29696 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:49.991429 29730 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.001880 29696 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.006789 29730 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.017776 29862 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.020323 29863 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.058682 29870 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.064644 29696 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:50.070679 27993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:36119, user_credentials={real_user=slave}} blocked reactor thread for 53353.1us
I20240503 14:15:50.076440 29696 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.080811 29870 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.084007 29870 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.084650 29696 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.109174 29811 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.124264 29696 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:50.166589 29811 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:50.177043 29850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:50.177580 29850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:50.199662 29850 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 464.35 ms
time per row: 0.232175 ms
I20240503 14:15:50.831970 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.8:44583
--local_ip_for_outbound_sockets=127.10.145.8
--tserver_master_addrs=127.10.145.62:42867
--webserver_port=34263
--webserver_interface=127.10.145.8
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:15:51.244905 29952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:51.245250 29952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:51.245404 29952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:51.253933 29952 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:51.254096 29952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:51.254262 29952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.8
I20240503 14:15:51.265323 29952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.8:44583
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.10.145.8
--webserver_port=34263
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.8
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:51.266875 29952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:51.268826 29952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:51.283390 29960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:51.283543 29962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:51.283890 29959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:51.287299 29952 server_base.cc:1034] running on GCE node
I20240503 14:15:51.288465 29952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:51.289662 29952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:51.290875 29952 hybrid_clock.cc:648] HybridClock initialized: now 1714745751290814 us; error 62 us; skew 500 ppm
I20240503 14:15:51.291205 29952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:51.299873 29952 webserver.cc:460] Webserver started at http://127.10.145.8:34263/ using document root <none> and password file <none>
I20240503 14:15:51.300817 29952 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:51.300946 29952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:51.306951 29952 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 14:15:51.310424 29968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:51.312009 29952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:51.312211 29952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e938d875af584f1f88a2d4b50c589268"
format_stamp: "Formatted at 2024-05-03 14:15:33 on dist-test-slave-v549"
I20240503 14:15:51.313167 29952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:51.347815 29952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:51.349004 29952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:51.349424 29952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:51.351063 29952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:51.352970 29952 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:51.353060 29952 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:51.353171 29952 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:51.353221 29952 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:51.393378 29952 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.8:44583
I20240503 14:15:51.393437 30080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.8:44583 every 8 connection(s)
I20240503 14:15:51.395464 29952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:15:51.397541 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 29952
I20240503 14:15:51.398293 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.9:39057
--local_ip_for_outbound_sockets=127.10.145.9
--tserver_master_addrs=127.10.145.62:42867
--webserver_port=46745
--webserver_interface=127.10.145.9
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:51.411883 30081 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:51.412245 30081 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:51.413173 30081 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:51.415192 27513 ts_manager.cc:194] Re-registered known tserver with Master: e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583)
I20240503 14:15:51.416311 27513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.8:45629
W20240503 14:15:51.801807 30085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:51.802134 30085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:51.802233 30085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:51.811039 30085 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:51.811182 30085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:51.811303 30085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.9
I20240503 14:15:51.821329 30085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.9:39057
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.10.145.9
--webserver_port=46745
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.9
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:51.823107 30085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:51.825383 30085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:51.838841 30091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:51.839107 30094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:51.839406 30092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:51.845329 30085 server_base.cc:1034] running on GCE node
I20240503 14:15:51.846086 30085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:51.847188 30085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:51.848438 30085 hybrid_clock.cc:648] HybridClock initialized: now 1714745751848373 us; error 64 us; skew 500 ppm
I20240503 14:15:51.848840 30085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:51.851593 30085 webserver.cc:460] Webserver started at http://127.10.145.9:46745/ using document root <none> and password file <none>
I20240503 14:15:51.852733 30085 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:51.852908 30085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:51.859225 30085 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20240503 14:15:51.862663 30101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:51.864231 30085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:51.864427 30085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "54dab61f1a204c0da4f56f98b3fd2690"
format_stamp: "Formatted at 2024-05-03 14:15:34 on dist-test-slave-v549"
I20240503 14:15:51.865345 30085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:51.896692 30085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:51.897794 30085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:51.898077 30085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:51.899091 30085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:51.900943 30085 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:51.901027 30085 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:51.901156 30085 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:51.901207 30085 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:51.942095 30085 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.9:39057
I20240503 14:15:51.942126 30213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.9:39057 every 8 connection(s)
I20240503 14:15:51.943969 30085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 14:15:51.950699 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 30085
I20240503 14:15:51.951529 10820 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
/tmp/dist-test-taskFZBpI6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.145.10:40989
--local_ip_for_outbound_sockets=127.10.145.10
--tserver_master_addrs=127.10.145.62:42867
--webserver_port=44195
--webserver_interface=127.10.145.10
--builtin_ntp_servers=127.10.145.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:15:51.960023 30214 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:51.960464 30214 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:51.961607 30214 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:51.963304 27513 ts_manager.cc:194] Re-registered known tserver with Master: 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057)
I20240503 14:15:51.964242 27513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.9:51255
W20240503 14:15:52.358278 30218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:52.358847 30218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:52.359019 30218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:52.367028 30218 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:15:52.367208 30218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:52.367384 30218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.145.10
I20240503 14:15:52.378374 30218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.145.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.145.10:40989
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.10.145.10
--webserver_port=44195
--tserver_master_addrs=127.10.145.62:42867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.145.10
--log_dir=/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:52.380172 30218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:52.382524 30218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:52.398770 30227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:52.399689 30228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:52.403611 30230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:52.404220 30218 server_base.cc:1034] running on GCE node
I20240503 14:15:52.404915 30218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:52.406080 30218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:52.407318 30218 hybrid_clock.cc:648] HybridClock initialized: now 1714745752407267 us; error 57 us; skew 500 ppm
I20240503 14:15:52.407651 30218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:52.410094 30218 webserver.cc:460] Webserver started at http://127.10.145.10:44195/ using document root <none> and password file <none>
I20240503 14:15:52.411052 30218 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:52.411190 30218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:52.417785 30218 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:52.419904 30081 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
I20240503 14:15:52.421449 30236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:52.423122 30218 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:15:52.423321 30218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "01173ed91fd341e3a4a4599118167e52"
format_stamp: "Formatted at 2024-05-03 14:15:34 on dist-test-slave-v549"
I20240503 14:15:52.424520 30218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:52.449842 30218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:52.450906 30218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:52.451237 30218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:52.452476 30218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:52.454356 30218 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:52.454440 30218 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:52.454561 30218 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:52.454609 30218 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:52.491808 30218 rpc_server.cc:307] RPC server started. Bound to: 127.10.145.10:40989
I20240503 14:15:52.491847 30348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.145.10:40989 every 8 connection(s)
I20240503 14:15:52.493616 30218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240503 14:15:52.497740 10820 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFZBpI6/build/asan/bin/kudu as pid 30218
I20240503 14:15:52.510110 30349 heartbeater.cc:347] Connected to a master server at 127.10.145.62:42867
I20240503 14:15:52.510527 30349 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:52.511725 30349 heartbeater.cc:511] Master 127.10.145.62:42867 requested a full tablet report, sending...
I20240503 14:15:52.513638 27513 ts_manager.cc:194] Re-registered known tserver with Master: 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989)
I20240503 14:15:52.514803 27513 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.145.10:35219
I20240503 14:15:52.967412 30214 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
I20240503 14:15:53.162245 27642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:53.164404 28299 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:53.184549 28430 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:53.214241 30284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.215405 30149 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.257125 28035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:53.271975 27904 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:53.288250 30016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.292029 27772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:53.297674 28168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:53.518740 30349 heartbeater.cc:503] Master 127.10.145.62:42867 was elected leader, sending a full tablet report...
I20240503 14:15:53.571632 30284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.574761 28035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:53.589203 27642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:53.633460 28430 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:53.648376 27904 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:53.654744 30149 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.668973 28299 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:53.685576 27772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:53.737802 28168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:53.758795 30016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.034320 30284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.040798 30149 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.041527 28035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:54.048998 27642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:54.054745 28430 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:54.065608 28299 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:54.074275 27772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:54.080345 30016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.094175 27904 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:54.141433 28168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:54.298512 28057 consensus_queue.cc:237] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:54.302279 27662 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.302557 27794 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.302706 28451 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.303557 29868 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.303929 30507 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.304232 28320 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.304674 27922 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.304754 28187 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.305788 29868 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.306126 30506 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.306596 30508 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.306993 30497 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.315030 29868 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.315795 28320 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.316711 27794 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.321825 28187 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.321873 28451 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.323909 27513 catalog_manager.cc:5484] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.335147 27922 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.336164 27662 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.352722 28057 consensus_queue.cc:237] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:54.352840 27794 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:54.358911 28320 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.359443 28055 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.359684 30496 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.360364 28187 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.361168 28186 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.362944 27662 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.363914 28451 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.364415 28450 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.365689 30220 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.366379 30527 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.367136 30220 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.368368 27662 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.369096 28319 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.369014 30506 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.370306 27922 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.370967 30506 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.371372 29868 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.376063 30527 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.378095 29868 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.382191 28319 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.385499 27922 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.386118 27793 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.387041 30220 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.388512 30497 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.389099 30497 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.389731 30497 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.390589 28187 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.393414 30220 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.395293 28318 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.396250 27924 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.401315 28057 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.402235 27521 catalog_manager.cc:5484] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.404831 27793 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.405251 28450 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.407507 27516 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.412621 28187 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
W20240503 14:15:54.415115 27993 consensus_peers.cc:485] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b178a32f4b24249aeae16cada4a3d56. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:54.414959 27661 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.416491 27508 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6b178a32f4b24249aeae16cada4a3d56 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:54.418737 27662 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.419107 28451 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.419626 27924 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.432386 27793 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:54.436887 27661 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.438616 30527 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.440256 28450 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.448474 30496 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.462323 28057 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.462613 28187 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.462982 27924 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.463313 30536 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:15:54.464802 30496 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:15:54.465549 30220 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.466645 28319 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.468192 28186 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:54.468722 30220 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.472932 30536 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.474093 28319 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.476544 27508 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d84acfe4f36f4193970fd2230229a41f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:54.477473 28450 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.477893 27662 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.478399 27517 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:54.478857 28187 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
W20240503 14:15:54.484340 27730 consensus_peers.cc:485] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d84acfe4f36f4193970fd2230229a41f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:54.484622 27922 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.487342 28057 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:54.498078 27793 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.498090 27662 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.503674 30543 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.504115 30546 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.505574 28319 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.505761 28450 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.506299 30545 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.507735 27924 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.508697 28057 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.506739 30548 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20240503 14:15:54.511137 30545 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.511488 30547 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.540822 30545 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.542520 28450 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.542456 28318 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.546021 28055 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.550697 27517 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:54.550765 27924 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.554136 27662 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.567430 28186 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:54.571055 28450 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.571859 28055 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.571859 28318 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.572088 30545 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.572456 30548 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.572997 30548 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.573261 27924 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.574105 27794 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.577157 27662 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.577782 27793 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.582242 30543 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.585485 30545 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.585912 27662 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.601970 27517 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.603937 27508 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12dae8a392374b91be378ec431838686 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:54.604725 28450 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.605221 28318 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.615267 27924 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:54.631534 28055 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.631801 28318 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.631922 28450 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.632588 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.632629 30545 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.632841 28186 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.633162 30548 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.634724 30355 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.634500 27924 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.635298 30355 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.636091 27662 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.636449 27794 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.641785 28056 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.644593 30355 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.645273 30355 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.645792 30355 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.654460 27794 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
W20240503 14:15:54.677613 28124 consensus_peers.cc:485] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12dae8a392374b91be378ec431838686. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:54.699622 27861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:35261, user_credentials={real_user=slave}} blocked reactor thread for 60237.2us
I20240503 14:15:54.702162 28187 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.705936 28318 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.700731 30355 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.710996 28055 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.711134 27517 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.717942 28450 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.720541 27792 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.728945 27662 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:54.732286 27924 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:54.739025 28318 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.740428 28055 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.740872 27661 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.741006 27792 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.741639 28187 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.741696 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.742197 30355 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.742452 30566 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.742758 28450 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.742838 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.743273 30566 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.744163 30355 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.748389 30567 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.749034 27661 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.752871 27508 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e895331729d3400384f20aa3895540de with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:54.754097 28450 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.755026 27517 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:54.757371 28318 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.770897 28055 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
W20240503 14:15:54.774291 27862 consensus_peers.cc:485] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e895331729d3400384f20aa3895540de. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:54.776185 28187 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.781175 27924 consensus_queue.cc:237] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:54.782222 27792 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:54.784416 28450 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.784550 27661 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.785014 27793 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.785106 30562 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.785336 28055 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.785754 30577 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.786078 30567 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.786252 28318 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.786813 30562 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.787145 30567 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.788810 28187 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.789916 30577 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.792829 30577 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.793610 27661 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.795562 28187 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.800458 27792 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.803256 27513 catalog_manager.cc:5484] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.812523 28450 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.816390 28319 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.818009 28057 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.828145 27924 consensus_queue.cc:237] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:54.831866 28057 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.832927 28318 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.833039 28187 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.834540 30355 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.835213 30355 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.838745 30567 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.841674 27661 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.842682 30592 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.842854 28450 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.843866 30567 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.846330 27792 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.848651 30355 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.849203 30355 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
W20240503 14:15:54.850853 27862 consensus_peers.cc:485] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae969579ee545c9838d3c26e4abd66f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:54.851970 28186 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:54.854197 28057 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:54.854564 27661 consensus_queue.cc:237] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:54.854622 27792 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:54.858084 27517 catalog_manager.cc:5484] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:54.859773 28451 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:54.863435 27792 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.864367 30498 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.864475 28450 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.865168 27508 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8ae969579ee545c9838d3c26e4abd66f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:54.865514 30498 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.867247 27662 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:54.867795 28187 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.868176 28057 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.868569 30499 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.869303 30499 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.869940 28318 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:54.870726 28319 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.871563 30499 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.880239 27924 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.881058 30499 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.891027 30499 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.895191 28055 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.900346 28319 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.901379 27517 catalog_manager.cc:5484] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.905581 27924 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.909859 28187 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.920449 30601 ts_tablet_manager.cc:902] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859)
I20240503 14:15:54.927381 30601 tablet_copy_client.cc:308] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:38859
I20240503 14:15:54.929695 28450 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.940223 28319 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:54.942164 27792 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:54.948273 27814 tablet_copy_service.cc:140] P 74318b6270b64efaa79867167fc08bbb: Received BeginTabletCopySession request for tablet d84acfe4f36f4193970fd2230229a41f from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:41641)
I20240503 14:15:54.948664 27814 tablet_copy_service.cc:161] P 74318b6270b64efaa79867167fc08bbb: Beginning new tablet copy session on tablet d84acfe4f36f4193970fd2230229a41f from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:41641: session id = 01173ed91fd341e3a4a4599118167e52-d84acfe4f36f4193970fd2230229a41f
I20240503 14:15:54.960278 27814 tablet_copy_source_session.cc:205] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:54.963189 30619 ts_tablet_manager.cc:902] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:15:54.974462 30621 ts_tablet_manager.cc:902] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605)
I20240503 14:15:54.975657 30601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84acfe4f36f4193970fd2230229a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:54.975780 27792 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.976774 28450 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.977428 30544 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.981940 27659 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.981698 27661 consensus_queue.cc:237] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:54.982864 30225 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.983556 30225 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.985358 27924 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.986783 28318 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.986896 30225 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.987494 28451 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.988456 28057 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.988678 30617 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.989066 28186 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:54.990005 30613 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:54.992131 28186 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.993337 30225 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:54.997764 28057 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:54.998374 30544 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:55.000643 30601 tablet_copy_client.cc:783] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:55.001055 30601 tablet_copy_client.cc:647] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:55.004750 30601 tablet_copy_client.cc:523] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:55.006584 27922 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:55.009858 30613 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:55.010903 30617 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:55.011803 30617 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:55.015126 30617 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:55.015640 27922 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:55.020483 27507 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 56dad6fc0ba0440aa4660030b1bb9267 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:55.020300 28055 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:55.023370 30544 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:55.026235 28451 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:55.028264 28187 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:55.030436 27517 catalog_manager.cc:5484] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:55.031914 27520 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:55.033951 30601 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:15:55.035140 30621 tablet_copy_client.cc:308] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.5:35605
I20240503 14:15:55.036140 30619 tablet_copy_client.cc:308] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:45951
I20240503 14:15:55.044423 27792 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:55.059063 28319 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:55.063848 28451 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:55.064728 30225 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:55.068091 27792 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:55.071592 30638 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:55.078294 28187 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:55.079622 28077 tablet_copy_service.cc:140] P 078f8eff84c94dde9705ad1330922a11: Received BeginTabletCopySession request for tablet 6b178a32f4b24249aeae16cada4a3d56 from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:38405)
I20240503 14:15:55.079905 28077 tablet_copy_service.cc:161] P 078f8eff84c94dde9705ad1330922a11: Beginning new tablet copy session on tablet 6b178a32f4b24249aeae16cada4a3d56 from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:38405: session id = 01173ed91fd341e3a4a4599118167e52-6b178a32f4b24249aeae16cada4a3d56
I20240503 14:15:55.079463 27659 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
W20240503 14:15:55.080199 27598 consensus_peers.cc:485] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56dad6fc0ba0440aa4660030b1bb9267. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:55.084146 27662 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:55.084300 27922 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:55.082623 30643 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.003s
I20240503 14:15:55.085511 30639 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:55.085755 30638 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:55.085754 28057 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:55.086059 28077 tablet_copy_source_session.cc:205] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:55.086380 28187 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:55.086939 27922 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:55.088341 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b178a32f4b24249aeae16cada4a3d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:55.090571 30638 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:55.091323 30638 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:55.098469 30619 tablet_copy_client.cc:783] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:55.098731 30619 tablet_copy_client.cc:647] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:55.103586 30619 tablet_copy_client.cc:523] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:55.104125 27792 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:55.104782 30498 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:55.107141 30619 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:15:55.106609 27792 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:55.108379 28187 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:55.111784 28319 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:55.114693 27507 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet add941da26824cdda7489b5da990e879 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:55.117460 27793 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:55.117493 28450 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:55.118232 27517 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:55.133085 28057 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:55.133102 27661 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:55.147815 28057 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:55.154120 30601 log.cc:824] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:55.163544 27922 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
W20240503 14:15:55.165588 28256 consensus_peers.cc:485] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: add941da26824cdda7489b5da990e879. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:55.166220 28208 tablet_copy_service.cc:140] P c91f285b8f4f4a27bcdd0ada303c1422: Received BeginTabletCopySession request for tablet 12dae8a392374b91be378ec431838686 from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:35369)
I20240503 14:15:55.166527 28208 tablet_copy_service.cc:161] P c91f285b8f4f4a27bcdd0ada303c1422: Beginning new tablet copy session on tablet 12dae8a392374b91be378ec431838686 from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:35369: session id = 54dab61f1a204c0da4f56f98b3fd2690-12dae8a392374b91be378ec431838686
I20240503 14:15:55.180984 30649 ts_tablet_manager.cc:902] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:55.186023 28208 tablet_copy_source_session.cc:205] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:55.198599 30621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dae8a392374b91be378ec431838686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:55.210433 28450 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:55.244691 30621 tablet_copy_client.cc:783] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:55.245218 30621 tablet_copy_client.cc:647] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:55.248215 30621 tablet_copy_client.cc:523] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:55.249953 30649 tablet_copy_client.cc:308] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:36119
I20240503 14:15:55.254187 30621 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:15:55.260316 27946 tablet_copy_service.cc:140] P a242f99f54b4494893064cfe3030b4ad: Received BeginTabletCopySession request for tablet e895331729d3400384f20aa3895540de from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:36319)
I20240503 14:15:55.260643 27946 tablet_copy_service.cc:161] P a242f99f54b4494893064cfe3030b4ad: Beginning new tablet copy session on tablet e895331729d3400384f20aa3895540de from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:36319: session id = 54dab61f1a204c0da4f56f98b3fd2690-e895331729d3400384f20aa3895540de
I20240503 14:15:55.266894 27946 tablet_copy_source_session.cc:205] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:55.270617 30649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e895331729d3400384f20aa3895540de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:55.281472 30649 tablet_copy_client.cc:783] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:55.281800 30649 tablet_copy_client.cc:647] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:55.284217 30649 tablet_copy_client.cc:523] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:55.290964 30649 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:15:55.297051 30621 log.cc:824] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:55.307197 30619 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:55.307798 30619 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:15:55.308357 30619 ts_tablet_manager.cc:1367] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.201s user 0.060s sys 0.008s
I20240503 14:15:55.315636 30619 raft_consensus.cc:348] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:55.316287 30619 raft_consensus.cc:729] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:15:55.317104 30619 consensus_queue.cc:260] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:55.340946 30619 ts_tablet_manager.cc:1398] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.032s user 0.008s sys 0.003s
I20240503 14:15:55.345026 28077 tablet_copy_service.cc:342] P 078f8eff84c94dde9705ad1330922a11: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-6b178a32f4b24249aeae16cada4a3d56 received from {username='slave'} at 127.10.145.10:38405
I20240503 14:15:55.345239 28077 tablet_copy_service.cc:433] P 078f8eff84c94dde9705ad1330922a11: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-6b178a32f4b24249aeae16cada4a3d56 on tablet 6b178a32f4b24249aeae16cada4a3d56 with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:15:55.356771 30304 raft_consensus.cc:1203] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:55.371733 30601 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:55.372223 30601 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:15:55.372543 30601 ts_tablet_manager.cc:1367] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.339s user 0.056s sys 0.019s
I20240503 14:15:55.373661 30601 raft_consensus.cc:348] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:55.374001 30601 raft_consensus.cc:729] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:15:55.374307 30601 consensus_queue.cc:260] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:55.375646 30601 ts_tablet_manager.cc:1398] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:55.377246 27814 tablet_copy_service.cc:342] P 74318b6270b64efaa79867167fc08bbb: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-d84acfe4f36f4193970fd2230229a41f received from {username='slave'} at 127.10.145.10:41641
I20240503 14:15:55.377496 27814 tablet_copy_service.cc:433] P 74318b6270b64efaa79867167fc08bbb: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-d84acfe4f36f4193970fd2230229a41f on tablet d84acfe4f36f4193970fd2230229a41f with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:15:55.416618 30649 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:55.417079 30649 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:15:55.417603 30649 ts_tablet_manager.cc:1367] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.127s user 0.047s sys 0.009s
I20240503 14:15:55.422648 30621 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:55.424103 30649 raft_consensus.cc:348] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:55.424787 30621 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:15:55.424839 30649 raft_consensus.cc:729] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:15:55.425088 30621 ts_tablet_manager.cc:1367] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.171s user 0.047s sys 0.019s
I20240503 14:15:55.425508 30649 consensus_queue.cc:260] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:55.428263 30621 raft_consensus.cc:348] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:55.428592 30621 raft_consensus.cc:729] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:15:55.428840 30621 consensus_queue.cc:260] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:55.429980 30621 ts_tablet_manager.cc:1398] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:15:55.430002 30649 ts_tablet_manager.cc:1398] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20240503 14:15:55.431238 28208 tablet_copy_service.cc:342] P c91f285b8f4f4a27bcdd0ada303c1422: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-12dae8a392374b91be378ec431838686 received from {username='slave'} at 127.10.145.9:35369
I20240503 14:15:55.431412 28208 tablet_copy_service.cc:433] P c91f285b8f4f4a27bcdd0ada303c1422: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-12dae8a392374b91be378ec431838686 on tablet 12dae8a392374b91be378ec431838686 with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:15:55.431778 27946 tablet_copy_service.cc:342] P a242f99f54b4494893064cfe3030b4ad: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-e895331729d3400384f20aa3895540de received from {username='slave'} at 127.10.145.9:36319
I20240503 14:15:55.431950 27946 tablet_copy_service.cc:433] P a242f99f54b4494893064cfe3030b4ad: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-e895331729d3400384f20aa3895540de on tablet e895331729d3400384f20aa3895540de with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:15:55.450183 30684 ts_tablet_manager.cc:902] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:15:55.451508 30601 ts_tablet_manager.cc:902] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:15:55.459712 30601 tablet_copy_client.cc:308] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:35261
I20240503 14:15:55.461444 30684 tablet_copy_client.cc:308] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:36119
I20240503 14:15:55.469244 27682 tablet_copy_service.cc:140] P 7046cbd82d634976b53fe0cef5788f18: Received BeginTabletCopySession request for tablet 56dad6fc0ba0440aa4660030b1bb9267 from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:60537)
I20240503 14:15:55.469549 27682 tablet_copy_service.cc:161] P 7046cbd82d634976b53fe0cef5788f18: Beginning new tablet copy session on tablet 56dad6fc0ba0440aa4660030b1bb9267 from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:60537: session id = 01173ed91fd341e3a4a4599118167e52-56dad6fc0ba0440aa4660030b1bb9267
I20240503 14:15:55.472890 27682 tablet_copy_source_session.cc:205] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:55.474861 30601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dad6fc0ba0440aa4660030b1bb9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:55.480356 30601 tablet_copy_client.cc:783] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:55.480475 27946 tablet_copy_service.cc:140] P a242f99f54b4494893064cfe3030b4ad: Received BeginTabletCopySession request for tablet 8ae969579ee545c9838d3c26e4abd66f from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:60517)
I20240503 14:15:55.480675 30601 tablet_copy_client.cc:647] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:55.480664 27946 tablet_copy_service.cc:161] P a242f99f54b4494893064cfe3030b4ad: Beginning new tablet copy session on tablet 8ae969579ee545c9838d3c26e4abd66f from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:60517: session id = e938d875af584f1f88a2d4b50c589268-8ae969579ee545c9838d3c26e4abd66f
I20240503 14:15:55.482843 30601 tablet_copy_client.cc:523] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:55.483358 27946 tablet_copy_source_session.cc:205] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:55.490530 30601 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:15:55.492101 30684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae969579ee545c9838d3c26e4abd66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:55.496212 30304 raft_consensus.cc:1203] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:55.512826 30684 tablet_copy_client.cc:783] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:55.513196 30684 tablet_copy_client.cc:647] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:55.517462 30684 tablet_copy_client.cc:523] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:55.522310 30684 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:15:55.528836 30601 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:55.529235 30601 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:15:55.529553 30601 ts_tablet_manager.cc:1367] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.039s user 0.029s sys 0.008s
I20240503 14:15:55.530342 30601 raft_consensus.cc:348] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:55.530612 30601 raft_consensus.cc:729] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:15:55.530854 30601 consensus_queue.cc:260] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:55.532570 30601 ts_tablet_manager.cc:1398] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:15:55.533769 27682 tablet_copy_service.cc:342] P 7046cbd82d634976b53fe0cef5788f18: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-56dad6fc0ba0440aa4660030b1bb9267 received from {username='slave'} at 127.10.145.10:60537
I20240503 14:15:55.533965 27682 tablet_copy_service.cc:433] P 7046cbd82d634976b53fe0cef5788f18: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-56dad6fc0ba0440aa4660030b1bb9267 on tablet 56dad6fc0ba0440aa4660030b1bb9267 with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:15:55.536623 30684 log.cc:824] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:55.572011 30684 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:55.572696 30684 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:15:55.573264 30684 ts_tablet_manager.cc:1367] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.051s user 0.046s sys 0.001s
I20240503 14:15:55.573956 30168 raft_consensus.cc:1203] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:55.580595 30684 raft_consensus.cc:348] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:55.581108 30684 raft_consensus.cc:729] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:15:55.581663 30684 consensus_queue.cc:260] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:55.583395 30684 ts_tablet_manager.cc:1398] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240503 14:15:55.584457 27946 tablet_copy_service.cc:342] P a242f99f54b4494893064cfe3030b4ad: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-8ae969579ee545c9838d3c26e4abd66f received from {username='slave'} at 127.10.145.8:60517
I20240503 14:15:55.584594 27946 tablet_copy_service.cc:433] P a242f99f54b4494893064cfe3030b4ad: ending tablet copy session e938d875af584f1f88a2d4b50c589268-8ae969579ee545c9838d3c26e4abd66f on tablet 8ae969579ee545c9838d3c26e4abd66f with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:15:55.621007 30621 ts_tablet_manager.cc:902] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:15:55.634406 30621 tablet_copy_client.cc:308] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36073
I20240503 14:15:55.641952 28340 tablet_copy_service.cc:140] P acb93936f2f14e628eecb05d7d1f4d91: Received BeginTabletCopySession request for tablet add941da26824cdda7489b5da990e879 from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:41081)
I20240503 14:15:55.642226 28340 tablet_copy_service.cc:161] P acb93936f2f14e628eecb05d7d1f4d91: Beginning new tablet copy session on tablet add941da26824cdda7489b5da990e879 from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:41081: session id = 54dab61f1a204c0da4f56f98b3fd2690-add941da26824cdda7489b5da990e879
I20240503 14:15:55.645376 28340 tablet_copy_source_session.cc:205] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:55.647473 30621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add941da26824cdda7489b5da990e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:55.652808 30621 tablet_copy_client.cc:783] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:55.653079 30621 tablet_copy_client.cc:647] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:55.654999 30621 tablet_copy_client.cc:523] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:55.657994 30621 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:15:55.690749 30621 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:55.691231 30621 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:15:55.691569 30621 ts_tablet_manager.cc:1367] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.034s user 0.030s sys 0.002s
I20240503 14:15:55.692500 30621 raft_consensus.cc:348] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:55.692890 30621 raft_consensus.cc:729] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:15:55.693145 30621 consensus_queue.cc:260] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:55.694087 30621 ts_tablet_manager.cc:1398] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:15:55.695248 28340 tablet_copy_service.cc:342] P acb93936f2f14e628eecb05d7d1f4d91: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-add941da26824cdda7489b5da990e879 received from {username='slave'} at 127.10.145.9:41081
I20240503 14:15:55.695430 28340 tablet_copy_service.cc:433] P acb93936f2f14e628eecb05d7d1f4d91: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-add941da26824cdda7489b5da990e879 on tablet add941da26824cdda7489b5da990e879 with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:15:55.734113 30168 raft_consensus.cc:1203] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:55.858510 30535 raft_consensus.cc:1050] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:15:55.859647 30535 consensus_queue.cc:237] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:15:55.862948 30304 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.863009 27922 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.863386 27792 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.863869 28450 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.864324 27659 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.864507 28319 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.864691 30750 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.865199 30618 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.865582 30751 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.865885 28187 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.866122 30618 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.866536 30535 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.879905 30752 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.881583 30752 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.884164 30618 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.885834 27922 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.888681 27792 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.889451 30036 raft_consensus.cc:1203] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:55.893705 27517 catalog_manager.cc:5484] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.896111 28450 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.896157 28319 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.899986 30304 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.907727 27659 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.916620 28187 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.933359 28057 consensus_queue.cc:237] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:15:55.936048 27922 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:55.936180 30304 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:55.936374 27792 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:55.936690 30752 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:55.936853 28187 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:55.937685 27659 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:55.937670 28450 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:55.937701 30750 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:55.938237 30750 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:55.938519 30535 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:55.938762 30618 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:55.939198 30752 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:55.942813 30750 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.946220 28187 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.946365 28450 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.948951 30538 raft_consensus.cc:1050] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:15:55.950408 30538 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:15:55.952787 30304 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.953032 27518 catalog_manager.cc:5484] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.954964 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b178a32f4b24249aeae16cada4a3d56 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:55.955701 27922 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.949646 27792 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.965114 28057 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.965309 28319 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.965852 30764 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.966691 30704 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.965981 27659 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:55.970960 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b178a32f4b24249aeae16cada4a3d56 with delete_type TABLET_DATA_TOMBSTONED (TS acb93936f2f14e628eecb05d7d1f4d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39840
I20240503 14:15:55.971999 28450 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.973222 30704 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.976481 30768 tablet_replica.cc:330] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91: stopping tablet replica
I20240503 14:15:55.977198 30768 raft_consensus.cc:2229] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:55.978587 30768 raft_consensus.cc:2258] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:55.982182 30768 ts_tablet_manager.cc:1875] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:55.989189 28187 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.990031 30765 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.990540 27922 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.990767 30304 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.991456 30764 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.992036 30764 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.995190 27661 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:55.996685 30764 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:55.999788 30545 raft_consensus.cc:1050] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:15:56.005182 30545 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:15:56.007762 30764 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.012524 28319 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.013324 27922 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.014797 28187 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.019727 27518 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.020704 28451 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.026444 30168 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.031117 30548 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.032104 27659 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.032791 30548 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.036784 30303 raft_consensus.cc:1203] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:56.040154 27922 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.041150 30548 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.042126 28057 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.058154 30304 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.064612 28319 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.064936 27793 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.065423 30779 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.066280 28451 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.067649 30543 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.068238 30543 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.069710 30168 raft_consensus.cc:1203] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:56.069870 30768 ts_tablet_manager.cc:1888] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:56.070067 30768 log.cc:1196] T 6b178a32f4b24249aeae16cada4a3d56 P acb93936f2f14e628eecb05d7d1f4d91: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/6b178a32f4b24249aeae16cada4a3d56
I20240503 14:15:56.073182 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet 6b178a32f4b24249aeae16cada4a3d56 (table rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]) successfully deleted
I20240503 14:15:56.074143 28057 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.073638 30779 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.074802 30543 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.074800 28319 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.079198 27662 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.089771 27922 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.090991 27513 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.103977 27662 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.104830 30168 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.106127 28450 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.116756 27792 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:15:56.120878 28450 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.121474 30538 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.121989 28187 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.122655 30538 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.124615 28319 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.125313 30764 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.125841 27661 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.126401 27923 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.126533 30764 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.127126 30764 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.130296 30764 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.133980 27923 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.140448 27516 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.142297 30304 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.141557 28319 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.142925 30777 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.143661 27792 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.144701 30303 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.149122 28057 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.153461 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet d84acfe4f36f4193970fd2230229a41f with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:15:56.158762 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d84acfe4f36f4193970fd2230229a41f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:56.160187 28187 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.168864 27662 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.174278 28450 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.176017 30789 tablet_replica.cc:330] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:15:56.176971 30789 raft_consensus.cc:2229] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:56.177510 30789 raft_consensus.cc:2258] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:56.179656 30789 ts_tablet_manager.cc:1875] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:56.188439 30789 ts_tablet_manager.cc:1888] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:56.188717 30789 log.cc:1196] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/d84acfe4f36f4193970fd2230229a41f
I20240503 14:15:56.189853 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet d84acfe4f36f4193970fd2230229a41f (table rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]) successfully deleted
I20240503 14:15:56.338598 30355 raft_consensus.cc:1050] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:15:56.339738 30355 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:15:56.343482 28319 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.343765 28057 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.343812 28187 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.344285 28451 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.344424 30567 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.344995 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.345597 30567 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.346526 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.353188 27662 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.353240 27792 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.353327 30168 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.353981 30355 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.354358 30824 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.354604 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.355381 30824 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.356158 28055 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.363948 28451 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.365017 27521 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.370414 30168 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.373481 28187 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.373692 28319 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.383339 27792 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.384884 27662 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.388861 27922 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:15:56.392436 28055 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.392892 30168 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.393533 30823 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.394106 28187 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.394148 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.395046 27662 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.395259 28451 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.395821 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.396648 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.397133 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.397679 27792 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.398252 30562 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.400228 30355 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.401003 28187 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.406935 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e895331729d3400384f20aa3895540de with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:56.406948 28451 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.408087 27792 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.408754 28055 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.410602 30168 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.413082 27661 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.413645 27513 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:56.419394 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet e895331729d3400384f20aa3895540de with delete_type TABLET_DATA_TOMBSTONED (TS acb93936f2f14e628eecb05d7d1f4d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39840
I20240503 14:15:56.419817 30768 tablet_replica.cc:330] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: stopping tablet replica
I20240503 14:15:56.420019 30768 raft_consensus.cc:2229] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:56.420279 30768 raft_consensus.cc:2258] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:56.421337 30768 ts_tablet_manager.cc:1875] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:56.425885 30768 ts_tablet_manager.cc:1888] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:56.426002 30768 log.cc:1196] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/e895331729d3400384f20aa3895540de
I20240503 14:15:56.426605 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet e895331729d3400384f20aa3895540de (table rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]) successfully deleted
I20240503 14:15:56.441051 30824 raft_consensus.cc:1050] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:15:56.442049 30824 consensus_queue.cc:237] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:15:56.445472 27662 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.445750 30036 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.446033 30562 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.446306 28187 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.446316 28319 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.446879 30823 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.446652 28055 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.447218 30355 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.447834 28451 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.448442 27792 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.449074 30824 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240503 14:15:56.449409 30833 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.449751 30567 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.450417 30562 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.455443 30638 raft_consensus.cc:1050] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:15:56.456883 30638 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:15:56.464437 27922 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.465128 30728 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.467106 30562 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.469269 28319 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.470258 30168 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.470858 30728 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.471721 27792 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.472410 30728 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.473636 28187 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.477943 27520 catalog_manager.cc:5484] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.481026 30036 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.482939 27662 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.484599 27661 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.485323 28451 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.485321 30643 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.486017 30643 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.497110 28057 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.497681 28187 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.497826 30638 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.498319 30643 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.512759 30643 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.515185 27792 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.517755 28451 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.520617 27659 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.521641 28450 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.521816 27520 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:56.521770 30168 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.525581 27791 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.531761 27922 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.534559 27923 consensus_queue.cc:237] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:15:56.535981 28057 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.540978 30036 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.541544 27662 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.541965 28319 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.542793 28449 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.544180 30823 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.544785 30823 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.545323 30823 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.548614 30567 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.549026 28186 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.544912 28055 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.551537 27793 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.553644 30355 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.554206 30355 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.558470 28186 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.557266 30355 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.564765 28449 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.571637 27513 catalog_manager.cc:5484] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.575421 28188 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.577929 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae969579ee545c9838d3c26e4abd66f with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:15:56.578330 30789 tablet_replica.cc:330] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:15:56.578536 30789 raft_consensus.cc:2229] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:56.578806 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ae969579ee545c9838d3c26e4abd66f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:56.578969 30789 raft_consensus.cc:2258] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:56.579612 27659 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.581161 28319 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.585623 28318 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:15:56.587220 27793 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.589891 27923 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.590517 30856 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.591881 30168 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.593184 30789 ts_tablet_manager.cc:1875] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:56.593554 30856 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.598765 27791 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.599403 30643 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.610509 30638 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.617617 28449 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.618376 30855 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.618881 27659 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.619807 30728 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.621501 27659 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.621714 28449 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.625564 27923 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.625752 30168 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.633309 28188 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.634920 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet add941da26824cdda7489b5da990e879 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:56.636270 27521 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.640573 30728 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.641902 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet add941da26824cdda7489b5da990e879 with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:15:56.642535 27791 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.642704 28188 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.612427 30036 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:56.668479 30647 raft_consensus.cc:1050] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:15:56.669966 30647 consensus_queue.cc:237] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:15:56.672626 30789 ts_tablet_manager.cc:1888] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:56.672884 30789 log.cc:1196] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/8ae969579ee545c9838d3c26e4abd66f
I20240503 14:15:56.674338 30789 tablet_replica.cc:330] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:15:56.674422 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet 8ae969579ee545c9838d3c26e4abd66f (table rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]) successfully deleted
I20240503 14:15:56.689059 28188 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:15:56.692508 30789 raft_consensus.cc:2229] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:56.693893 27659 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.694676 30168 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.694612 30543 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.695322 30543 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.696514 27923 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.698575 30545 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.701442 30789 raft_consensus.cc:2258] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:56.703255 30789 ts_tablet_manager.cc:1875] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:56.702814 30543 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.703826 27922 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.705305 27923 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.705843 30852 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.706557 30304 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.707955 28055 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.708250 30852 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.708534 30545 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.710405 28318 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.710532 28055 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.711215 28449 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.711773 27793 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.712194 28057 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.713562 30545 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.714726 27791 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.714746 30854 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.715734 28318 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.717590 30854 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.717800 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12dae8a392374b91be378ec431838686 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:56.718221 30854 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.724117 27513 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 6 to 7, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.729485 28430 tablet_service.cc:1514] Processing DeleteTablet for tablet 12dae8a392374b91be378ec431838686 with delete_type TABLET_DATA_TOMBSTONED (TS 9231e715a4e342a197d6c5456ba6b399 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60466
I20240503 14:15:56.732720 30545 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.734421 28186 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.747915 30852 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.747587 27659 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.749121 30168 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.751194 28319 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:56.757851 27791 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:56.758949 30852 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:56.762984 30864 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.769068 27793 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.771122 30873 tablet_replica.cc:330] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399: stopping tablet replica
I20240503 14:15:56.771654 30873 raft_consensus.cc:2229] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:56.770208 28057 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.774574 27924 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.776551 28449 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.777123 27513 catalog_manager.cc:5484] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:56.784269 28188 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.797814 30789 ts_tablet_manager.cc:1888] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:56.797748 30304 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.799006 30789 log.cc:1196] T add941da26824cdda7489b5da990e879 P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/add941da26824cdda7489b5da990e879
I20240503 14:15:56.799466 30873 raft_consensus.cc:2258] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:56.803395 30873 ts_tablet_manager.cc:1875] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:56.804037 28318 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.808197 27659 consensus_queue.cc:237] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:15:56.812013 28449 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.812013 30304 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.812676 30858 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.812857 27793 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.813074 30852 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.814855 30852 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.815438 27924 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.816948 30647 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.818338 28319 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:56.821272 28186 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.820859 30858 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.821870 30852 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.823386 28188 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.823990 27924 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.830121 27521 catalog_manager.cc:5484] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.831423 28449 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.833169 30304 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.834977 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet 56dad6fc0ba0440aa4660030b1bb9267 with delete_type TABLET_DATA_TOMBSTONED (TS acb93936f2f14e628eecb05d7d1f4d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39840
I20240503 14:15:56.835812 27793 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.836992 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56dad6fc0ba0440aa4660030b1bb9267 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:56.839563 30768 tablet_replica.cc:330] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91: stopping tablet replica
I20240503 14:15:56.839789 30768 raft_consensus.cc:2229] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:56.840314 30768 raft_consensus.cc:2258] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:56.841485 30768 ts_tablet_manager.cc:1875] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:56.820545 28055 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:56.844430 30862 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:56.846681 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet add941da26824cdda7489b5da990e879 (table rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]) successfully deleted
I20240503 14:15:56.846534 28055 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:56.847251 30873 ts_tablet_manager.cc:1888] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:56.847497 30873 log.cc:1196] T 12dae8a392374b91be378ec431838686 P 9231e715a4e342a197d6c5456ba6b399: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/wals/12dae8a392374b91be378ec431838686
I20240503 14:15:56.849071 27506 catalog_manager.cc:4830] TS 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): tablet 12dae8a392374b91be378ec431838686 (table rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]) successfully deleted
I20240503 14:15:56.851543 27659 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.851824 27924 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.852301 27793 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.852278 30643 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.853322 28449 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.850914 30768 ts_tablet_manager.cc:1888] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:56.853920 30728 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.854048 30768 log.cc:1196] T 56dad6fc0ba0440aa4660030b1bb9267 P acb93936f2f14e628eecb05d7d1f4d91: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/56dad6fc0ba0440aa4660030b1bb9267
I20240503 14:15:56.854769 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet 56dad6fc0ba0440aa4660030b1bb9267 (table rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]) successfully deleted
I20240503 14:15:56.855069 30643 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.855569 30643 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.858773 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet 56dad6fc0ba0440aa4660030b1bb9267 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:39840
I20240503 14:15:56.859494 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet 56dad6fc0ba0440aa4660030b1bb9267 (table rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]) successfully deleted
I20240503 14:15:56.860303 28188 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.863834 28057 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.864614 30856 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.864930 30643 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.864152 30855 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:56.865104 27659 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:56.873270 28188 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:56.876641 27521 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:56.880892 27793 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:56.890113 28319 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:56.892033 28449 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:56.894187 27793 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:56.894757 30855 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:56.901330 27662 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:56.903095 30855 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:56.911854 28450 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:56.914592 30855 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:56.922605 28186 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:56.930601 27922 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:56.930959 30855 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:56.932145 30856 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:56.939499 28057 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:56.940004 27507 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 233ee592f6e843c8a68a5170e0f9d66d with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:56.941035 27659 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:56.941275 28449 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:56.947480 27520 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:56.948537 30856 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.949642 27793 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:56.950217 27924 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
W20240503 14:15:56.957913 28256 consensus_peers.cc:485] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 233ee592f6e843c8a68a5170e0f9d66d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:56.960956 28188 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:56.962543 27924 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:56.971895 28055 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:56.972784 30855 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:56.974926 28055 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:56.978205 27659 consensus_queue.cc:237] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:56.981539 27793 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.981600 28449 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.982335 30854 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.982885 28055 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.982884 30862 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.983011 28188 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.983842 30647 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.984154 30862 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.987229 27924 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.987911 30862 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.987977 28319 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.988530 30647 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.992007 30647 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:56.994745 28319 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:56.995252 28188 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.001161 27793 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.001885 27521 catalog_manager.cc:5484] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.002250 28449 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.006909 28055 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.008603 27924 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.016522 27659 consensus_queue.cc:237] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:57.022469 28450 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.023151 28319 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.023156 28188 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.023252 27791 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:15:57.023304 27598 consensus_peers.cc:485] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77469d6a6610402ca96598bb695224d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:57.023588 28055 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.024048 30862 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.024401 30858 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.024909 30858 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.025672 27924 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.025799 30863 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.026118 30864 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.026546 30858 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.030493 30864 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:57.032557 28055 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:57.032748 28188 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:57.034651 27507 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 77469d6a6610402ca96598bb695224d9 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:57.037552 28319 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:57.037834 28450 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:57.039067 27793 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:57.039584 27518 catalog_manager.cc:5484] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:57.044996 27924 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:15:57.345186 30939 ts_tablet_manager.cc:902] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:15:57.346767 30939 tablet_copy_client.cc:308] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36073
I20240503 14:15:57.354275 28340 tablet_copy_service.cc:140] P acb93936f2f14e628eecb05d7d1f4d91: Received BeginTabletCopySession request for tablet 233ee592f6e843c8a68a5170e0f9d66d from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:54229)
I20240503 14:15:57.354488 28340 tablet_copy_service.cc:161] P acb93936f2f14e628eecb05d7d1f4d91: Beginning new tablet copy session on tablet 233ee592f6e843c8a68a5170e0f9d66d from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:54229: session id = e938d875af584f1f88a2d4b50c589268-233ee592f6e843c8a68a5170e0f9d66d
I20240503 14:15:57.356412 28340 tablet_copy_source_session.cc:205] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:57.357842 30939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ee592f6e843c8a68a5170e0f9d66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:57.362828 30939 tablet_copy_client.cc:783] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:57.363094 30939 tablet_copy_client.cc:647] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:57.364818 30939 tablet_copy_client.cc:523] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:57.367484 30939 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:15:57.406572 30939 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:57.407207 30939 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:15:57.407601 30939 ts_tablet_manager.cc:1367] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.040s user 0.033s sys 0.007s
I20240503 14:15:57.409552 30939 raft_consensus.cc:348] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:57.410027 30939 raft_consensus.cc:729] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:15:57.410286 30939 consensus_queue.cc:260] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:57.418787 30939 ts_tablet_manager.cc:1398] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.011s user 0.000s sys 0.010s
I20240503 14:15:57.419656 28340 tablet_copy_service.cc:342] P acb93936f2f14e628eecb05d7d1f4d91: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-233ee592f6e843c8a68a5170e0f9d66d received from {username='slave'} at 127.10.145.8:54229
I20240503 14:15:57.419812 28340 tablet_copy_service.cc:433] P acb93936f2f14e628eecb05d7d1f4d91: ending tablet copy session e938d875af584f1f88a2d4b50c589268-233ee592f6e843c8a68a5170e0f9d66d on tablet 233ee592f6e843c8a68a5170e0f9d66d with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:15:57.426849 28450 consensus_queue.cc:237] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:57.433255 28319 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.433395 28055 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.433540 28186 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.433971 30454 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.434540 30955 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.435699 27793 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.436458 30954 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.438688 27924 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.438822 30801 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.439172 27659 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.439373 30954 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.441901 30954 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.452404 30801 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.453620 27924 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.455396 27659 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.459327 28188 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.460651 28319 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.461136 27521 catalog_manager.cc:5484] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:57.471613 28450 consensus_queue.cc:237] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:57.474622 28055 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.475095 27793 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.481828 27659 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.482931 30955 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.485980 28319 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.486536 30955 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.495355 27793 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.495368 27924 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.496457 28186 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.497527 28057 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.498296 30955 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.499182 30955 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.501148 30955 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.501878 30955 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.503563 30801 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:57.506182 28057 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:57.508567 27506 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 44df52bbf3534802bf7217fb5cdb4326 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:57.510401 27662 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:57.512868 27793 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:57.513895 27518 catalog_manager.cc:5484] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:57.514927 27924 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:57.516544 28319 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:57.528663 28188 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
W20240503 14:15:57.537496 28387 consensus_peers.cc:485] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44df52bbf3534802bf7217fb5cdb4326. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:57.545868 27662 consensus_queue.cc:237] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:57.558857 27793 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.559135 27924 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.559581 28450 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.559687 30854 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.560209 30858 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.560956 30864 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.562304 28188 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.562757 28057 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.564071 30647 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.564669 30647 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.566334 28319 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.568312 30861 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.571858 30854 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.572896 28188 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.577888 28450 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.578053 28318 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.578773 27791 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.584120 27518 catalog_manager.cc:5484] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:57.593565 30972 ts_tablet_manager.cc:902] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:15:57.598026 27662 consensus_queue.cc:237] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:57.607630 28319 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.610256 28450 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.611413 30647 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.611935 30647 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.625511 30972 tablet_copy_client.cc:308] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:35261
I20240503 14:15:57.626053 28057 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.626451 28188 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.626691 27682 tablet_copy_service.cc:140] P 7046cbd82d634976b53fe0cef5788f18: Received BeginTabletCopySession request for tablet 77469d6a6610402ca96598bb695224d9 from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:60537)
I20240503 14:15:57.626878 27682 tablet_copy_service.cc:161] P 7046cbd82d634976b53fe0cef5788f18: Beginning new tablet copy session on tablet 77469d6a6610402ca96598bb695224d9 from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:60537: session id = 01173ed91fd341e3a4a4599118167e52-77469d6a6610402ca96598bb695224d9
I20240503 14:15:57.627179 27924 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.627583 27791 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.629819 27682 tablet_copy_source_session.cc:205] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:57.632256 30972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77469d6a6610402ca96598bb695224d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:57.635514 30861 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.636441 30861 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.637130 30861 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.637861 30861 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.639506 28057 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:57.640390 30972 tablet_copy_client.cc:783] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:57.640673 30972 tablet_copy_client.cc:647] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:57.642710 30972 tablet_copy_client.cc:523] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:57.650545 30972 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:15:57.650696 27924 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
W20240503 14:15:57.653585 27598 consensus_peers.cc:485] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbb645fc3c5c422b9e3fabba8ce12873. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:57.654865 30647 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.656610 28188 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.656729 28057 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.659585 28450 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.663940 27521 catalog_manager.cc:5484] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.667696 28318 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.669925 27507 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bbb645fc3c5c422b9e3fabba8ce12873 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:57.672658 27791 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.674886 27924 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.690922 28450 consensus_queue.cc:237] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:57.704556 27791 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.705933 28318 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.705981 28057 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.707026 30954 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.707180 28188 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.707753 30954 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.707849 27924 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.710978 30454 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.711521 30454 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.711935 30454 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.731801 30801 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.733683 28188 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.742383 27521 catalog_manager.cc:5484] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.743500 27924 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.749114 30972 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:57.749696 30972 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:15:57.750056 30972 ts_tablet_manager.cc:1367] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.100s user 0.051s sys 0.001s
I20240503 14:15:57.750139 28057 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.751312 28318 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.751852 30972 raft_consensus.cc:348] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:57.752171 30972 raft_consensus.cc:729] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:15:57.752408 30972 consensus_queue.cc:260] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:15:57.752918 27662 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.754267 30801 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.759692 27662 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.763156 27791 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.767755 28450 consensus_queue.cc:237] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:57.772763 27659 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.773895 30955 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.775136 30972 ts_tablet_manager.cc:1398] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.025s user 0.010s sys 0.011s
I20240503 14:15:57.776273 27924 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.776854 30801 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.777312 27682 tablet_copy_service.cc:342] P 7046cbd82d634976b53fe0cef5788f18: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-77469d6a6610402ca96598bb695224d9 received from {username='slave'} at 127.10.145.10:60537
I20240503 14:15:57.777456 27682 tablet_copy_service.cc:433] P 7046cbd82d634976b53fe0cef5788f18: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-77469d6a6610402ca96598bb695224d9 on tablet 77469d6a6610402ca96598bb695224d9 with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:15:57.781759 28188 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.782967 28318 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.784121 30954 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.784493 28057 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.787573 27793 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.788501 30454 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.788936 30454 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:15:57.789705 28387 consensus_peers.cc:485] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48a05a802960425383984c379e238682. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:57.790115 30454 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.792577 30801 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.793437 28318 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.794749 28057 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.796705 27506 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 48a05a802960425383984c379e238682 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:57.797750 27659 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.801379 27521 catalog_manager.cc:5484] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.802572 28188 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.803478 27791 consensus_queue.cc:237] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:57.806452 27924 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.808032 28186 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.809345 28318 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.809324 30874 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.810065 28450 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.810050 30874 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.810678 30874 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.811363 28055 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.815377 27793 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.817955 30874 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.821072 27924 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.822117 27659 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.822117 30874 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.822728 30874 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.828023 31014 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.831220 28450 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.837908 27518 catalog_manager.cc:5484] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.837860 27924 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.845057 27793 consensus_queue.cc:237] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:57.847366 28055 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.848444 27924 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.848940 28450 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.849471 31014 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.852063 28188 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.853838 28318 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.863790 28319 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.864699 31014 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.868309 31014 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.871762 28055 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.873451 28186 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.873939 31014 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.877133 30874 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.879706 31016 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.880746 28055 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.880949 28188 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.888597 27521 catalog_manager.cc:5484] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.902601 27508 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:57.902451 27924 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.905071 28450 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.905121 28318 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
W20240503 14:15:57.911266 27730 consensus_peers.cc:485] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06b7e707cec54bc8bc5e649c3b37dfb5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:57.914896 27659 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:57.919258 31024 ts_tablet_manager.cc:902] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:57.919493 27662 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:57.920445 31016 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:57.922112 27662 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:15:57.929560 31024 tablet_copy_client.cc:308] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.7:43675
I20240503 14:15:57.936718 28471 tablet_copy_service.cc:140] P 9231e715a4e342a197d6c5456ba6b399: Received BeginTabletCopySession request for tablet 44df52bbf3534802bf7217fb5cdb4326 from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:54935)
I20240503 14:15:57.936977 28471 tablet_copy_service.cc:161] P 9231e715a4e342a197d6c5456ba6b399: Beginning new tablet copy session on tablet 44df52bbf3534802bf7217fb5cdb4326 from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:54935: session id = 54dab61f1a204c0da4f56f98b3fd2690-44df52bbf3534802bf7217fb5cdb4326
I20240503 14:15:57.940366 28471 tablet_copy_source_session.cc:205] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:57.947607 31024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44df52bbf3534802bf7217fb5cdb4326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:57.953622 30036 raft_consensus.cc:1203] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:57.953954 31024 tablet_copy_client.cc:783] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:57.954253 31024 tablet_copy_client.cc:647] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:57.959266 31024 tablet_copy_client.cc:523] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:57.962884 31024 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:15:58.002943 31024 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:58.003357 31024 tablet_bootstrap.cc:492] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:15:58.003615 31024 ts_tablet_manager.cc:1367] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.041s user 0.036s sys 0.001s
I20240503 14:15:58.004335 31024 raft_consensus.cc:348] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:58.004595 31024 raft_consensus.cc:729] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:15:58.004793 31024 consensus_queue.cc:260] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:58.005642 31024 ts_tablet_manager.cc:1398] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:15:58.006618 28471 tablet_copy_service.cc:342] P 9231e715a4e342a197d6c5456ba6b399: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-44df52bbf3534802bf7217fb5cdb4326 received from {username='slave'} at 127.10.145.9:54935
I20240503 14:15:58.006825 28471 tablet_copy_service.cc:433] P 9231e715a4e342a197d6c5456ba6b399: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-44df52bbf3534802bf7217fb5cdb4326 on tablet 44df52bbf3534802bf7217fb5cdb4326 with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:15:58.103488 31061 ts_tablet_manager.cc:902] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:15:58.111172 30304 raft_consensus.cc:1203] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:58.138691 31061 tablet_copy_client.cc:308] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:35261
I20240503 14:15:58.147308 27682 tablet_copy_service.cc:140] P 7046cbd82d634976b53fe0cef5788f18: Received BeginTabletCopySession request for tablet bbb645fc3c5c422b9e3fabba8ce12873 from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:47681)
I20240503 14:15:58.147516 27682 tablet_copy_service.cc:161] P 7046cbd82d634976b53fe0cef5788f18: Beginning new tablet copy session on tablet bbb645fc3c5c422b9e3fabba8ce12873 from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:47681: session id = e938d875af584f1f88a2d4b50c589268-bbb645fc3c5c422b9e3fabba8ce12873
I20240503 14:15:58.150673 27682 tablet_copy_source_session.cc:205] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:58.152432 31061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb645fc3c5c422b9e3fabba8ce12873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.159369 31061 tablet_copy_client.cc:783] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:58.159698 31061 tablet_copy_client.cc:647] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:58.161725 31061 tablet_copy_client.cc:523] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:58.164381 31061 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:15:58.198168 31061 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:58.198559 31061 tablet_bootstrap.cc:492] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:15:58.198812 31061 ts_tablet_manager.cc:1367] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.035s user 0.031s sys 0.004s
I20240503 14:15:58.199451 31061 raft_consensus.cc:348] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:58.199699 31061 raft_consensus.cc:729] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:15:58.199916 31061 consensus_queue.cc:260] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:58.200472 31061 ts_tablet_manager.cc:1398] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:58.201401 27682 tablet_copy_service.cc:342] P 7046cbd82d634976b53fe0cef5788f18: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-bbb645fc3c5c422b9e3fabba8ce12873 received from {username='slave'} at 127.10.145.8:47681
I20240503 14:15:58.201519 27682 tablet_copy_service.cc:433] P 7046cbd82d634976b53fe0cef5788f18: ending tablet copy session e938d875af584f1f88a2d4b50c589268-bbb645fc3c5c422b9e3fabba8ce12873 on tablet bbb645fc3c5c422b9e3fabba8ce12873 with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:15:58.253069 31061 ts_tablet_manager.cc:902] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:15:58.254350 31061 tablet_copy_client.cc:308] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.7:43675
I20240503 14:15:58.261183 28471 tablet_copy_service.cc:140] P 9231e715a4e342a197d6c5456ba6b399: Received BeginTabletCopySession request for tablet 48a05a802960425383984c379e238682 from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:59745)
I20240503 14:15:58.261428 28471 tablet_copy_service.cc:161] P 9231e715a4e342a197d6c5456ba6b399: Beginning new tablet copy session on tablet 48a05a802960425383984c379e238682 from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:59745: session id = e938d875af584f1f88a2d4b50c589268-48a05a802960425383984c379e238682
I20240503 14:15:58.264034 28471 tablet_copy_source_session.cc:205] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:58.265610 31061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a05a802960425383984c379e238682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.270534 31061 tablet_copy_client.cc:783] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:58.270968 31061 tablet_copy_client.cc:647] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:58.273056 31061 tablet_copy_client.cc:523] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:58.275835 31061 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:15:58.315332 31061 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:58.315757 31061 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:15:58.316066 31061 ts_tablet_manager.cc:1367] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.040s user 0.031s sys 0.007s
I20240503 14:15:58.316877 31061 raft_consensus.cc:348] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:58.317147 31061 raft_consensus.cc:729] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:15:58.317423 31061 consensus_queue.cc:260] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:58.318020 31061 ts_tablet_manager.cc:1398] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:15:58.318761 28471 tablet_copy_service.cc:342] P 9231e715a4e342a197d6c5456ba6b399: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-48a05a802960425383984c379e238682 received from {username='slave'} at 127.10.145.8:59745
I20240503 14:15:58.318907 28471 tablet_copy_service.cc:433] P 9231e715a4e342a197d6c5456ba6b399: ending tablet copy session e938d875af584f1f88a2d4b50c589268-48a05a802960425383984c379e238682 on tablet 48a05a802960425383984c379e238682 with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:15:58.405129 31061 ts_tablet_manager.cc:902] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859)
I20240503 14:15:58.409373 31061 tablet_copy_client.cc:308] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:38859
I20240503 14:15:58.415494 27814 tablet_copy_service.cc:140] P 74318b6270b64efaa79867167fc08bbb: Received BeginTabletCopySession request for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:51227)
I20240503 14:15:58.415656 27814 tablet_copy_service.cc:161] P 74318b6270b64efaa79867167fc08bbb: Beginning new tablet copy session on tablet 06b7e707cec54bc8bc5e649c3b37dfb5 from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:51227: session id = e938d875af584f1f88a2d4b50c589268-06b7e707cec54bc8bc5e649c3b37dfb5
I20240503 14:15:58.417624 27814 tablet_copy_source_session.cc:205] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:58.418916 31061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b7e707cec54bc8bc5e649c3b37dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.424296 31061 tablet_copy_client.cc:783] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:58.424554 31061 tablet_copy_client.cc:647] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:58.426359 31061 tablet_copy_client.cc:523] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:58.428977 31061 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:15:58.433676 30855 raft_consensus.cc:1050] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:15:58.434617 30855 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:15:58.439342 27924 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.439610 27662 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.440032 28186 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.440306 28055 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.440605 30643 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.441176 30643 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.441254 30036 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.441267 28450 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.443723 30856 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.444111 30643 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.444628 30856 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.455713 30856 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.456275 27791 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.458820 30855 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.478070 30856 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.480849 31061 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:58.481315 31061 tablet_bootstrap.cc:492] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:15:58.481578 31061 ts_tablet_manager.cc:1367] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.053s user 0.043s sys 0.006s
I20240503 14:15:58.482759 31061 raft_consensus.cc:348] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:58.483093 31061 raft_consensus.cc:729] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:15:58.482519 27791 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.482635 28055 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.483337 31061 consensus_queue.cc:260] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:15:58.484005 31061 ts_tablet_manager.cc:1398] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:58.484155 27662 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.484936 27814 tablet_copy_service.cc:342] P 74318b6270b64efaa79867167fc08bbb: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-06b7e707cec54bc8bc5e649c3b37dfb5 received from {username='slave'} at 127.10.145.8:51227
I20240503 14:15:58.485065 27814 tablet_copy_service.cc:433] P 74318b6270b64efaa79867167fc08bbb: ending tablet copy session e938d875af584f1f88a2d4b50c589268-06b7e707cec54bc8bc5e649c3b37dfb5 on tablet 06b7e707cec54bc8bc5e649c3b37dfb5 with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:15:58.489048 27518 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.494140 30036 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.495622 28186 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.499529 27924 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.505407 30168 raft_consensus.cc:1203] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:58.511574 28450 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.543844 28318 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:15:58.547331 30036 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.547427 28450 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.548173 28186 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.548260 27791 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.548323 27924 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.549034 31110 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.549229 27662 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.549479 31118 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.549975 30643 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.551525 30643 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.552343 30856 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.552439 30858 raft_consensus.cc:1050] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:15:58.553647 30858 consensus_queue.cc:237] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:15:58.557214 27924 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.557224 30304 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.558285 28450 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.559387 30036 raft_consensus.cc:1203] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:58.559671 27791 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.560194 30854 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.560818 30854 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.562427 30861 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.563141 30861 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.565569 31118 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.572438 28055 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.573792 28186 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.574615 30854 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.575466 30854 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.576022 31110 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.577145 30036 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.576920 27922 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.581697 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 233ee592f6e843c8a68a5170e0f9d66d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:58.584208 27521 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.588991 27662 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.590692 28450 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.594780 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet 233ee592f6e843c8a68a5170e0f9d66d with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:15:58.611120 31141 tablet_replica.cc:330] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
W20240503 14:15:58.611187 27596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:43675, user_credentials={real_user=slave}} blocked reactor thread for 49163.5us
I20240503 14:15:58.611546 31141 raft_consensus.cc:2229] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:58.611974 31141 raft_consensus.cc:2258] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:58.612192 28318 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:58.614967 30861 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:58.618793 31141 ts_tablet_manager.cc:1875] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:58.618525 28188 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.634964 30858 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.635550 28057 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.642031 28449 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.648319 27520 catalog_manager.cc:5484] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.649672 30304 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.648555 27791 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:58.654716 27924 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.654836 27793 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.655805 31141 ts_tablet_manager.cc:1888] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:58.656018 31141 log.cc:1196] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/233ee592f6e843c8a68a5170e0f9d66d
I20240503 14:15:58.658380 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet 233ee592f6e843c8a68a5170e0f9d66d (table rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]) successfully deleted
I20240503 14:15:58.663538 28186 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
W20240503 14:15:58.720965 27598 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:36119, user_credentials={real_user=slave}} blocked reactor thread for 47172.4us
W20240503 14:15:58.720448 27862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.2:38859, user_credentials={real_user=slave}} blocked reactor thread for 75015us
W20240503 14:15:58.725248 27861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.6:36073, user_credentials={real_user=slave}} blocked reactor thread for 49721.8us
I20240503 14:15:58.728500 27662 consensus_queue.cc:237] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:15:58.728226 28318 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.732116 30304 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.732508 27791 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.733351 28449 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.733806 31152 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.734158 28186 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.734376 31152 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.734827 31152 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.738651 31153 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.739708 27923 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.739933 28057 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:58.739634 30858 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.740336 30304 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.744050 31153 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.745401 28186 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.745853 27923 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.746436 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 77469d6a6610402ca96598bb695224d9 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:58.748798 27520 catalog_manager.cc:5484] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 reported cstate change: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.750375 28449 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.752830 27793 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.757221 30854 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:58.760375 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet 77469d6a6610402ca96598bb695224d9 with delete_type TABLET_DATA_TOMBSTONED (TS acb93936f2f14e628eecb05d7d1f4d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39840
I20240503 14:15:58.761082 31155 tablet_replica.cc:330] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91: stopping tablet replica
I20240503 14:15:58.760674 28055 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:15:58.761520 31155 raft_consensus.cc:2229] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:58.761865 31155 raft_consensus.cc:2258] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:58.762863 31155 ts_tablet_manager.cc:1875] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:58.770543 31155 ts_tablet_manager.cc:1888] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:58.770715 31155 log.cc:1196] T 77469d6a6610402ca96598bb695224d9 P acb93936f2f14e628eecb05d7d1f4d91: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/77469d6a6610402ca96598bb695224d9
I20240503 14:15:58.771570 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet 77469d6a6610402ca96598bb695224d9 (table rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]) successfully deleted
I20240503 14:15:58.783588 30036 raft_consensus.cc:1203] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:58.808732 30036 raft_consensus.cc:1203] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:15:59.018486 30854 raft_consensus.cc:1050] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:15:59.019577 30854 consensus_queue.cc:237] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:15:59.029763 28057 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:59.030197 27923 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.030539 28055 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.030890 30858 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.031347 27793 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.032042 30036 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.032739 28186 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.033300 28450 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.033546 31149 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.033972 28318 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.034193 31153 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.034570 31152 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.035537 31153 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.036855 31149 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.038199 28318 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.039084 28188 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.039242 27922 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.040269 28449 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.040215 30892 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.040931 27662 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.041201 27793 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.041821 30892 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.045022 31149 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.047127 30892 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.047695 30892 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.048197 30892 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.049643 31145 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.057052 30858 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.062628 28057 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.065742 27923 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.066074 28319 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.067875 27513 catalog_manager.cc:5484] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:59.070820 30036 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.079573 31145 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.085345 27793 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.088114 28450 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.091408 27662 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.096925 28186 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.106108 27520 catalog_manager.cc:5484] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.117503 27659 consensus_queue.cc:237] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:15:59.120807 31121 raft_consensus.cc:1050] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:15:59.121186 28318 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.121706 28186 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.122061 31121 consensus_queue.cc:237] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:15:59.122272 31153 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.123134 31153 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.123430 27793 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.124308 28450 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.124423 27923 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.124882 31150 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.125680 31149 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.126353 30036 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.125160 28187 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.126979 28451 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.130470 31149 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.132869 31150 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.134639 27923 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.137320 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bbb645fc3c5c422b9e3fabba8ce12873 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:59.138577 27793 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.141515 31149 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.141199 28187 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.139135 28318 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.143198 30035 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.138154 28319 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.146598 28451 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.148186 27512 catalog_manager.cc:5484] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
W20240503 14:15:59.166646 27992 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:35261, user_credentials={real_user=slave}} blocked reactor thread for 59140.3us
I20240503 14:15:59.168277 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet bbb645fc3c5c422b9e3fabba8ce12873 with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:15:59.173317 27923 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.174031 31123 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.181388 30168 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.181576 28188 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.181635 28320 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.181862 27791 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.182912 31005 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.183192 31123 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.183797 31123 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.193495 27662 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.194259 31005 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:15:59.204748 27993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:36119, user_credentials={real_user=slave}} blocked reactor thread for 101488us
I20240503 14:15:59.206466 27923 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.218042 31123 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.234620 28055 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.237318 28057 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:59.249593 31208 tablet_replica.cc:330] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:15:59.249945 31208 raft_consensus.cc:2229] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:59.250219 31208 raft_consensus.cc:2258] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:59.252207 31210 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.254261 28317 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.258448 31208 ts_tablet_manager.cc:1875] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:59.258445 28188 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.258518 27791 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.262017 27662 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.263761 27792 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.264504 31121 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.265585 31203 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240503 14:15:59.265547 27793 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.271081 28449 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.274658 31195 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:59.275749 27923 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.278484 31121 raft_consensus.cc:1050] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:15:59.279768 31121 consensus_queue.cc:237] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:15:59.290593 28317 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.291499 31195 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:59.268384 27512 catalog_manager.cc:5484] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.301777 31205 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.302531 28317 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.307350 31208 ts_tablet_manager.cc:1888] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:59.310343 31208 log.cc:1196] T bbb645fc3c5c422b9e3fabba8ce12873 P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/bbb645fc3c5c422b9e3fabba8ce12873
I20240503 14:15:59.309779 28449 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.312294 27508 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9f00380931064f71bc78626af72488f3 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:59.312597 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet bbb645fc3c5c422b9e3fabba8ce12873 (table rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]) successfully deleted
I20240503 14:15:59.313274 27662 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.314198 27513 catalog_manager.cc:5484] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:59.324492 30035 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.324731 28319 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.325524 31123 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.327980 31123 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.332276 27791 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.333375 27659 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.337595 31121 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.338042 31123 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.340906 28188 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.341657 31123 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.343712 27923 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.345748 31123 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.351784 31123 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.355968 30035 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.359338 27791 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.360821 27659 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.364382 27923 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.367815 27513 catalog_manager.cc:5484] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 reported cstate change: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.367406 28319 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.369571 28188 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.381204 30168 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.384163 27659 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.390183 31207 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.391494 27923 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.394673 27791 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.407568 28188 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.410095 28057 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.411991 31195 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:59.412904 31210 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.415132 27923 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.416150 31230 raft_consensus.cc:1050] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:15:59.417971 31195 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:15:59.423522 27993 consensus_peers.cc:485] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f00380931064f71bc78626af72488f3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:59.423308 28188 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.427649 27659 consensus_queue.cc:237] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:15:59.431622 31230 consensus_queue.cc:237] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:15:59.432303 28057 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.440183 28055 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.442967 28317 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.443367 28319 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.443506 30035 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.443661 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.444098 31177 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.444705 31177 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.444811 28188 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.446813 31177 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.447289 28449 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.447377 28450 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.448618 31233 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.449043 31153 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.453214 28057 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.453851 28187 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.454648 27922 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.455880 31150 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.456533 31150 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.457023 31150 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.465584 31150 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.466917 27922 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.467224 28057 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.471982 28319 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.473112 28186 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.473431 27520 catalog_manager.cc:5484] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.482504 27922 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.483098 27659 consensus_queue.cc:237] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:59.483739 31230 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.485011 27662 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.454422 27791 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.488945 28317 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.489579 31233 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.491555 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:59.491744 28451 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.492570 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.493963 28057 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:15:59.494400 28451 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.494861 31233 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:59.505255 27791 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.506354 27924 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.507809 28186 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.508971 28055 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.509976 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.512614 27791 raft_consensus.cc:858] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }:
New config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
W20240503 14:15:59.512792 27791 raft_consensus.cc:1565] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 5' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }:
New config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:59.517939 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:59.518945 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:59.519821 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:59.528247 31246 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.534842 28317 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.538594 28451 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.542402 31153 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.543617 30035 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.543989 28188 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.544685 27512 catalog_manager.cc:5484] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:59.547304 27507 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f15a6ff6acc242258172e585b1ea34c6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:15:59.550511 28317 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.551683 28057 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.557030 27512 catalog_manager.cc:5484] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:59.559891 28186 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.559098 27923 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.566049 28055 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.568020 28449 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.569579 27922 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.580519 27662 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.588565 27793 consensus_queue.cc:237] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
W20240503 14:15:59.589275 27598 consensus_peers.cc:485] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f15a6ff6acc242258172e585b1ea34c6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:59.593325 30035 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.593961 28319 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.594094 28188 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.594729 31177 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.594813 27923 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.595317 31177 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.596547 31246 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.597668 31230 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.599334 27662 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.600208 31230 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.602999 28449 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.603529 31246 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.605823 31233 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.605743 27791 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:15:59.607540 30035 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.608322 28449 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.610060 27662 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.614197 27516 catalog_manager.cc:5484] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.625640 27793 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:15:59.627430 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:15:59.627832 31208 tablet_replica.cc:330] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:15:59.628032 31208 raft_consensus.cc:2229] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:59.628273 31208 raft_consensus.cc:2258] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:59.630069 31208 ts_tablet_manager.cc:1875] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:59.630808 28186 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.634723 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 06b7e707cec54bc8bc5e649c3b37dfb5 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:59.636863 27923 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.641783 31208 ts_tablet_manager.cc:1888] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:59.641947 31208 log.cc:1196] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/06b7e707cec54bc8bc5e649c3b37dfb5
I20240503 14:15:59.643231 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet 06b7e707cec54bc8bc5e649c3b37dfb5 (table rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]) successfully deleted
I20240503 14:15:59.662441 28317 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.745030 28449 consensus_queue.cc:237] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:15:59.747846 28057 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.747895 27922 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.748210 27662 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.748524 30168 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.748795 31220 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.749087 28186 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.749388 31220 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.749912 27793 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.750409 31220 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.750663 31210 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.751188 31123 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.752918 31220 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.754813 27922 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.755360 31005 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.756011 27662 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.758631 30168 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.759537 27506 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44df52bbf3534802bf7217fb5cdb4326 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:59.760634 28186 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.760872 28057 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.761754 27516 catalog_manager.cc:5484] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:15:59.767800 27793 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:15:59.774683 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet 44df52bbf3534802bf7217fb5cdb4326 with delete_type TABLET_DATA_TOMBSTONED (TS acb93936f2f14e628eecb05d7d1f4d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39840
I20240503 14:15:59.775866 31288 tablet_replica.cc:330] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91: stopping tablet replica
I20240503 14:15:59.776152 31288 raft_consensus.cc:2229] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:59.776469 31288 raft_consensus.cc:2258] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:59.778340 31288 ts_tablet_manager.cc:1875] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:59.782618 31288 ts_tablet_manager.cc:1888] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:59.782730 31288 log.cc:1196] T 44df52bbf3534802bf7217fb5cdb4326 P acb93936f2f14e628eecb05d7d1f4d91: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/44df52bbf3534802bf7217fb5cdb4326
I20240503 14:15:59.783471 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet 44df52bbf3534802bf7217fb5cdb4326 (table rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]) successfully deleted
I20240503 14:15:59.849896 28449 consensus_queue.cc:237] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:15:59.852332 30035 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.852712 28186 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.853204 27793 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.853255 28057 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.853343 27659 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.853957 31220 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.854096 27922 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:15:59.854506 31290 ts_tablet_manager.cc:902] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:15:59.854630 31220 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.855391 31220 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.856074 31220 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.856828 31220 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.857322 31210 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:15:59.860946 31121 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.864642 27506 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48a05a802960425383984c379e238682 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:15:59.865295 31290 tablet_copy_client.cc:308] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:45951
I20240503 14:15:59.865532 27659 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.870203 27512 catalog_manager.cc:5484] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:15:59.874318 28057 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.874694 28186 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.875068 30035 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.878418 27793 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.879528 27922 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:15:59.883574 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a05a802960425383984c379e238682 with delete_type TABLET_DATA_TOMBSTONED (TS acb93936f2f14e628eecb05d7d1f4d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39840
I20240503 14:15:59.883960 31288 tablet_replica.cc:330] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: stopping tablet replica
I20240503 14:15:59.884122 31288 raft_consensus.cc:2229] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:59.884464 31288 raft_consensus.cc:2258] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:59.885466 31288 ts_tablet_manager.cc:1875] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:59.892308 31288 ts_tablet_manager.cc:1888] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:15:59.892449 31288 log.cc:1196] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/48a05a802960425383984c379e238682
I20240503 14:15:59.892658 28077 tablet_copy_service.cc:140] P 078f8eff84c94dde9705ad1330922a11: Received BeginTabletCopySession request for tablet 9f00380931064f71bc78626af72488f3 from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:57647)
I20240503 14:15:59.892854 28077 tablet_copy_service.cc:161] P 078f8eff84c94dde9705ad1330922a11: Beginning new tablet copy session on tablet 9f00380931064f71bc78626af72488f3 from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:57647: session id = 54dab61f1a204c0da4f56f98b3fd2690-9f00380931064f71bc78626af72488f3
I20240503 14:15:59.893748 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet 48a05a802960425383984c379e238682 (table rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]) successfully deleted
I20240503 14:15:59.895382 28077 tablet_copy_source_session.cc:205] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:59.897513 31290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f00380931064f71bc78626af72488f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.902036 31290 tablet_copy_client.cc:783] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:59.902285 31290 tablet_copy_client.cc:647] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:59.903859 31290 tablet_copy_client.cc:523] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:59.906476 31290 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:15:59.939379 31290 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:59.939862 31290 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:15:59.940172 31290 ts_tablet_manager.cc:1367] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.008s
I20240503 14:15:59.941126 31290 raft_consensus.cc:348] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:59.941530 31290 raft_consensus.cc:729] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:15:59.941814 31290 consensus_queue.cc:260] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:15:59.942456 31290 ts_tablet_manager.cc:1398] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:15:59.943411 28077 tablet_copy_service.cc:342] P 078f8eff84c94dde9705ad1330922a11: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-9f00380931064f71bc78626af72488f3 received from {username='slave'} at 127.10.145.9:57647
I20240503 14:15:59.943529 28077 tablet_copy_service.cc:433] P 078f8eff84c94dde9705ad1330922a11: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-9f00380931064f71bc78626af72488f3 on tablet 9f00380931064f71bc78626af72488f3 with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:15:59.952415 28449 consensus_queue.cc:237] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:15:59.955782 28057 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.955920 27922 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.956035 27793 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.956465 28186 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.956542 31121 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.957121 31121 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.957350 28317 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.957427 31210 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.957832 31123 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.958279 31121 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.958583 27659 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.962581 31121 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.972273 31121 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:59.973686 31290 ts_tablet_manager.cc:902] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:15:59.973116 27793 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:59.981184 27659 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:59.987552 28317 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:59.987705 27518 catalog_manager.cc:5484] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } }
I20240503 14:15:59.983915 28186 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:15:59.995810 28449 consensus_queue.cc:237] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:00.005031 28055 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.006095 28057 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.007550 27659 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.008144 31210 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.008489 27922 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.011090 28319 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.013319 31121 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.016157 27791 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.017863 28186 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.018785 31210 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.019237 31121 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.020668 31210 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.022568 31210 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.023129 27793 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.028138 27506 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:16:00.029851 31290 tablet_copy_client.cc:308] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:35261
I20240503 14:16:00.029655 27662 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.031112 27520 catalog_manager.cc:5484] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.032486 28186 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.033726 28317 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
W20240503 14:16:00.046483 28387 consensus_peers.cc:485] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89fb4ed8413f4eceb4348f9d216d3dd7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:00.048398 27682 tablet_copy_service.cc:140] P 7046cbd82d634976b53fe0cef5788f18: Received BeginTabletCopySession request for tablet f15a6ff6acc242258172e585b1ea34c6 from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:57721)
I20240503 14:16:00.048601 27682 tablet_copy_service.cc:161] P 7046cbd82d634976b53fe0cef5788f18: Beginning new tablet copy session on tablet f15a6ff6acc242258172e585b1ea34c6 from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:57721: session id = 54dab61f1a204c0da4f56f98b3fd2690-f15a6ff6acc242258172e585b1ea34c6
I20240503 14:16:00.050648 27924 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.051771 31005 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.052074 27682 tablet_copy_source_session.cc:205] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:00.053767 31290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15a6ff6acc242258172e585b1ea34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.054108 28055 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.054363 27922 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.061849 31290 tablet_copy_client.cc:783] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:00.062186 31290 tablet_copy_client.cc:647] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:00.067638 31290 tablet_copy_client.cc:523] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:00.071280 31290 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:16:00.079422 28449 consensus_queue.cc:237] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } }
I20240503 14:16:00.093823 28317 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.094475 28055 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.094552 31121 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.095193 31121 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.105019 28186 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.105294 27922 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.105991 31220 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.106586 31220 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.109249 27793 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.109823 27662 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.113724 31220 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.114279 31220 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.131985 31121 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:16:00.134943 31290 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:00.135409 31290 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:16:00.135694 31290 ts_tablet_manager.cc:1367] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.064s user 0.042s sys 0.001s
I20240503 14:16:00.136601 31290 raft_consensus.cc:348] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:00.136921 31290 raft_consensus.cc:729] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:16:00.137149 31290 consensus_queue.cc:260] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:00.137818 31290 ts_tablet_manager.cc:1398] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240503 14:16:00.140370 27793 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:16:00.146448 28317 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:16:00.150580 27682 tablet_copy_service.cc:342] P 7046cbd82d634976b53fe0cef5788f18: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-f15a6ff6acc242258172e585b1ea34c6 received from {username='slave'} at 127.10.145.9:57721
I20240503 14:16:00.150722 27682 tablet_copy_service.cc:433] P 7046cbd82d634976b53fe0cef5788f18: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-f15a6ff6acc242258172e585b1ea34c6 on tablet f15a6ff6acc242258172e585b1ea34c6 with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:16:00.150216 28055 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:16:00.155797 27516 catalog_manager.cc:5484] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } }
I20240503 14:16:00.163749 28449 consensus_queue.cc:237] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:00.169070 28055 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.169888 31121 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:16:00.171660 28387 consensus_peers.cc:485] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d9c369355ad47f4843da19dc3ae3cac. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:00.174263 28317 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.174890 31121 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.178973 27793 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.179493 31121 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.184223 31220 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:00.193491 27506 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0d9c369355ad47f4843da19dc3ae3cac with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:16:00.195626 27793 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:00.203898 27515 catalog_manager.cc:5484] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:00.212090 27659 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.213750 31121 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.215305 27662 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:16:00.219231 28055 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:00.219662 28317 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:00.225419 28187 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.226270 31332 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.228042 28187 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:16:00.237851 27922 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.239786 27662 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:00.240887 31332 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.242514 27922 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } }
I20240503 14:16:00.243814 28187 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:00.246609 27922 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:00.301760 30168 raft_consensus.cc:1203] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:00.428345 31361 ts_tablet_manager.cc:902] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:16:00.430289 31361 tablet_copy_client.cc:308] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.7:43675
I20240503 14:16:00.437443 28471 tablet_copy_service.cc:140] P 9231e715a4e342a197d6c5456ba6b399: Received BeginTabletCopySession request for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:50025)
I20240503 14:16:00.437633 28471 tablet_copy_service.cc:161] P 9231e715a4e342a197d6c5456ba6b399: Beginning new tablet copy session on tablet 89fb4ed8413f4eceb4348f9d216d3dd7 from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:50025: session id = 01173ed91fd341e3a4a4599118167e52-89fb4ed8413f4eceb4348f9d216d3dd7
I20240503 14:16:00.439831 28471 tablet_copy_source_session.cc:205] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:00.441236 31361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fb4ed8413f4eceb4348f9d216d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.446070 31361 tablet_copy_client.cc:783] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:00.446370 31361 tablet_copy_client.cc:647] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:00.448504 31361 tablet_copy_client.cc:523] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:00.451500 31361 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:16:00.474017 30168 raft_consensus.cc:1203] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:00.497582 31361 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:00.498065 31361 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:16:00.498347 31361 ts_tablet_manager.cc:1367] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.047s user 0.047s sys 0.000s
I20240503 14:16:00.499183 31361 raft_consensus.cc:348] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:00.499524 31361 raft_consensus.cc:729] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:16:00.499751 31361 consensus_queue.cc:260] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:00.507232 31361 ts_tablet_manager.cc:1398] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20240503 14:16:00.508656 28471 tablet_copy_service.cc:342] P 9231e715a4e342a197d6c5456ba6b399: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-89fb4ed8413f4eceb4348f9d216d3dd7 received from {username='slave'} at 127.10.145.10:50025
I20240503 14:16:00.508822 28471 tablet_copy_service.cc:433] P 9231e715a4e342a197d6c5456ba6b399: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-89fb4ed8413f4eceb4348f9d216d3dd7 on tablet 89fb4ed8413f4eceb4348f9d216d3dd7 with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:16:00.550995 28186 consensus_queue.cc:237] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:16:00.554025 28449 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.554090 27791 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.554453 28055 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.554952 28317 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.555286 27662 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.555330 27922 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.556020 31368 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.556401 31378 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:16:00.557382 31120 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.558055 31120 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.558331 31378 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.569830 31378 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.572875 31378 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.574694 27662 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.575554 28317 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.579048 28449 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.579159 27515 catalog_manager.cc:5484] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:00.579618 27922 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.580618 27791 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.580857 28055 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.598294 28186 consensus_queue.cc:237] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:00.603332 28317 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.603902 28451 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.604441 31120 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.605545 31379 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.613796 27662 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.614564 31379 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.616602 28055 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.622551 31368 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.630115 27508 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 29945558359248c6a8a4c68952cfcc81 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:16:00.630338 27922 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.628664 27662 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.631666 27791 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.632555 31120 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.632985 28317 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.634025 31120 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.633988 28451 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.639696 28055 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.638942 27515 catalog_manager.cc:5484] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:00.645002 31368 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.646054 27922 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.653481 27791 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
W20240503 14:16:00.663020 28124 consensus_peers.cc:485] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29945558359248c6a8a4c68952cfcc81. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:00.671418 28055 consensus_queue.cc:237] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:16:00.675323 27791 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.676432 31289 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.679937 28449 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.680208 27662 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.680569 31289 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.680876 27922 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.681156 31289 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.681797 31289 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.685139 28317 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.685153 28186 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.685822 31207 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.686412 31195 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.708663 31289 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.713604 28319 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.719753 27662 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.719797 27793 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.726024 27515 catalog_manager.cc:5484] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.739125 28186 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.741824 27922 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.758247 28451 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:00.762008 28055 consensus_queue.cc:237] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:00.765877 28187 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.766820 31403 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.767143 28451 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:16:00.767227 27993 consensus_peers.cc:485] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e9e6daafc14437faff996cfb40f6dfb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:00.767673 27922 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.767714 31195 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.768337 31195 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.769968 31411 ts_tablet_manager.cc:902] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:16:00.772401 31195 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.780478 27508 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0e9e6daafc14437faff996cfb40f6dfb with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:16:00.780910 28451 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.781318 28317 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.782045 31207 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.783774 28317 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.786657 27516 catalog_manager.cc:5484] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.787070 27662 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.787837 31409 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.787894 27922 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.791158 27793 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:00.791386 28187 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.792973 27662 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.799683 31409 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:00.801550 27793 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:00.813532 31411 tablet_copy_client.cc:308] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.7:43675
I20240503 14:16:00.815435 28471 tablet_copy_service.cc:140] P 9231e715a4e342a197d6c5456ba6b399: Received BeginTabletCopySession request for tablet 0d9c369355ad47f4843da19dc3ae3cac from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:59745)
I20240503 14:16:00.815672 28471 tablet_copy_service.cc:161] P 9231e715a4e342a197d6c5456ba6b399: Beginning new tablet copy session on tablet 0d9c369355ad47f4843da19dc3ae3cac from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:59745: session id = e938d875af584f1f88a2d4b50c589268-0d9c369355ad47f4843da19dc3ae3cac
I20240503 14:16:00.818957 28471 tablet_copy_source_session.cc:205] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:00.820457 31411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9c369355ad47f4843da19dc3ae3cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.826059 31411 tablet_copy_client.cc:783] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:00.826328 31411 tablet_copy_client.cc:647] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:00.828331 31411 tablet_copy_client.cc:523] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:00.832278 31411 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:16:00.861864 30304 raft_consensus.cc:1203] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:00.874267 31410 raft_consensus.cc:1050] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:16:00.874528 31411 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:00.874991 31411 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:16:00.875283 31411 ts_tablet_manager.cc:1367] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.043s user 0.029s sys 0.012s
I20240503 14:16:00.876001 31411 raft_consensus.cc:348] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:00.875770 31410 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:00.876336 31411 raft_consensus.cc:729] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:16:00.876554 31411 consensus_queue.cc:260] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:00.879273 31132 raft_consensus.cc:1050] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:16:00.879456 31411 ts_tablet_manager.cc:1398] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20240503 14:16:00.880249 28451 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.880349 27922 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.880822 28317 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.880831 28471 tablet_copy_service.cc:342] P 9231e715a4e342a197d6c5456ba6b399: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-0d9c369355ad47f4843da19dc3ae3cac received from {username='slave'} at 127.10.145.8:59745
I20240503 14:16:00.880638 31132 consensus_queue.cc:237] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:00.880949 28471 tablet_copy_service.cc:433] P 9231e715a4e342a197d6c5456ba6b399: ending tablet copy session e938d875af584f1f88a2d4b50c589268-0d9c369355ad47f4843da19dc3ae3cac on tablet 0d9c369355ad47f4843da19dc3ae3cac with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:16:00.881464 28187 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.881970 31289 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.882579 31289 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.882999 31409 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.883327 31195 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.886885 27662 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.889036 31409 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.900818 27791 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.901028 30168 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.912601 31195 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.913050 31432 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.918825 31432 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.919991 27791 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.926719 27516 catalog_manager.cc:5484] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.928602 28317 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.929430 27662 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.933452 28187 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.933928 28449 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.938336 28451 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.938956 30169 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.939879 31150 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.944757 28055 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:00.949467 27662 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:00.949829 28186 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:00.950114 31410 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:00.951534 28451 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:00.952879 28317 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.953441 31150 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.953758 28055 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.954371 31150 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.957736 27793 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:00.957751 31410 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:00.958683 31432 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:00.959120 31403 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:00.962615 30168 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:00.962934 28186 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.946960 27922 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.963969 27923 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.964144 31369 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.964882 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.965179 31410 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:00.969774 31432 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.970641 28187 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.974340 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9f00380931064f71bc78626af72488f3 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:00.976365 27924 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:00.975872 27659 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.977166 27520 catalog_manager.cc:5484] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:00.975682 27791 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.981165 31432 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:00.982980 27924 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.984066 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f00380931064f71bc78626af72488f3 with delete_type TABLET_DATA_TOMBSTONED (TS acb93936f2f14e628eecb05d7d1f4d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39840
I20240503 14:16:00.987355 31450 tablet_replica.cc:330] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: stopping tablet replica
I20240503 14:16:00.987663 31450 raft_consensus.cc:2229] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:00.988035 31450 raft_consensus.cc:2258] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:00.989499 31450 ts_tablet_manager.cc:1875] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:00.993945 30168 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.993924 28451 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:00.995497 27791 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:00.996217 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:00.999373 30169 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.000610 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.004467 31149 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.010421 31450 ts_tablet_manager.cc:1888] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:01.010650 31450 log.cc:1196] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/9f00380931064f71bc78626af72488f3
I20240503 14:16:01.011641 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet 9f00380931064f71bc78626af72488f3 (table rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]) successfully deleted
I20240503 14:16:01.012966 28451 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.008266 28319 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.019404 27520 catalog_manager.cc:5484] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.021658 28187 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.026402 27924 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.028124 27791 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.029464 30169 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.037081 28055 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.049724 27659 consensus_queue.cc:237] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:01.052830 27791 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.052845 28451 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.053122 28187 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.053732 27924 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.053922 30169 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.054385 31149 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.053350 28319 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.054734 31150 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.055025 31455 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.055352 31369 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.055634 31132 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.056164 31454 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.060561 31149 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.062578 27922 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.064535 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f15a6ff6acc242258172e585b1ea34c6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:01.066085 28451 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.068444 27516 catalog_manager.cc:5484] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:01.068928 30169 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.069648 28319 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.074651 27791 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.075946 28187 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:01.080570 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet f15a6ff6acc242258172e585b1ea34c6 with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:16:01.081588 31462 tablet_replica.cc:330] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:16:01.081841 31462 raft_consensus.cc:2229] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:01.082813 31462 raft_consensus.cc:2258] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:01.084213 31462 ts_tablet_manager.cc:1875] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:01.093430 31462 ts_tablet_manager.cc:1888] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:01.093595 31462 log.cc:1196] T f15a6ff6acc242258172e585b1ea34c6 P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/f15a6ff6acc242258172e585b1ea34c6
I20240503 14:16:01.094723 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet f15a6ff6acc242258172e585b1ea34c6 (table rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]) successfully deleted
I20240503 14:16:01.131742 31466 ts_tablet_manager.cc:902] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605)
I20240503 14:16:01.144915 31466 tablet_copy_client.cc:308] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.5:35605
I20240503 14:16:01.152520 28208 tablet_copy_service.cc:140] P c91f285b8f4f4a27bcdd0ada303c1422: Received BeginTabletCopySession request for tablet 29945558359248c6a8a4c68952cfcc81 from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:37207)
I20240503 14:16:01.152721 28208 tablet_copy_service.cc:161] P c91f285b8f4f4a27bcdd0ada303c1422: Beginning new tablet copy session on tablet 29945558359248c6a8a4c68952cfcc81 from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:37207: session id = 01173ed91fd341e3a4a4599118167e52-29945558359248c6a8a4c68952cfcc81
I20240503 14:16:01.155162 28208 tablet_copy_source_session.cc:205] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:01.156489 31466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29945558359248c6a8a4c68952cfcc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:01.161144 31466 tablet_copy_client.cc:783] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:01.161471 31466 tablet_copy_client.cc:647] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:01.163224 31466 tablet_copy_client.cc:523] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:01.165695 31466 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:16:01.184151 31474 ts_tablet_manager.cc:902] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:16:01.185845 31474 tablet_copy_client.cc:308] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:45951
I20240503 14:16:01.186599 28077 tablet_copy_service.cc:140] P 078f8eff84c94dde9705ad1330922a11: Received BeginTabletCopySession request for tablet 0e9e6daafc14437faff996cfb40f6dfb from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:38405)
I20240503 14:16:01.186753 28077 tablet_copy_service.cc:161] P 078f8eff84c94dde9705ad1330922a11: Beginning new tablet copy session on tablet 0e9e6daafc14437faff996cfb40f6dfb from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:38405: session id = 01173ed91fd341e3a4a4599118167e52-0e9e6daafc14437faff996cfb40f6dfb
I20240503 14:16:01.189142 28077 tablet_copy_source_session.cc:205] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:01.190632 31474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9e6daafc14437faff996cfb40f6dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:01.196245 31474 tablet_copy_client.cc:783] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:01.196506 31474 tablet_copy_client.cc:647] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:01.198534 31474 tablet_copy_client.cc:523] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:01.202498 31474 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:16:01.208424 31466 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:01.208914 31466 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:16:01.209192 31466 ts_tablet_manager.cc:1367] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.044s user 0.034s sys 0.003s
I20240503 14:16:01.210155 31466 raft_consensus.cc:348] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:01.210476 31466 raft_consensus.cc:729] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:16:01.210708 31466 consensus_queue.cc:260] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:01.212189 31466 ts_tablet_manager.cc:1398] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:16:01.216341 28208 tablet_copy_service.cc:342] P c91f285b8f4f4a27bcdd0ada303c1422: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-29945558359248c6a8a4c68952cfcc81 received from {username='slave'} at 127.10.145.10:37207
I20240503 14:16:01.216492 28208 tablet_copy_service.cc:433] P c91f285b8f4f4a27bcdd0ada303c1422: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-29945558359248c6a8a4c68952cfcc81 on tablet 29945558359248c6a8a4c68952cfcc81 with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:16:01.243034 31474 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:01.243433 31474 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:16:01.243690 31474 ts_tablet_manager.cc:1367] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.041s user 0.033s sys 0.005s
I20240503 14:16:01.244330 31474 raft_consensus.cc:348] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:01.244608 31474 raft_consensus.cc:729] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:16:01.244835 31474 consensus_queue.cc:260] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:01.245419 31474 ts_tablet_manager.cc:1398] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:16:01.246604 28077 tablet_copy_service.cc:342] P 078f8eff84c94dde9705ad1330922a11: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-0e9e6daafc14437faff996cfb40f6dfb received from {username='slave'} at 127.10.145.10:38405
I20240503 14:16:01.246758 28077 tablet_copy_service.cc:433] P 078f8eff84c94dde9705ad1330922a11: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-0e9e6daafc14437faff996cfb40f6dfb on tablet 0e9e6daafc14437faff996cfb40f6dfb with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:16:01.265416 30035 raft_consensus.cc:1203] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:01.331101 31210 raft_consensus.cc:1050] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:16:01.332291 31210 consensus_queue.cc:237] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:01.336695 30304 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.337931 27662 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.338306 31121 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.338706 31332 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.339177 27924 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.340636 27791 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.342355 31332 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.348465 31121 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.352217 28319 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.353011 31497 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.353452 28055 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.354108 28186 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.357993 31121 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.358623 31332 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.363478 31121 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.363979 28055 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.368913 30304 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.370167 27924 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.370707 27662 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.371528 28187 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.372922 27520 catalog_manager.cc:5484] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:01.376755 27791 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.377790 28319 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.382632 28451 consensus_queue.cc:237] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:01.386459 27662 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.387313 31497 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.387776 27924 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.389474 31497 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.390331 28186 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.390890 31497 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.394876 30303 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.397794 28317 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.399284 27793 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.402056 31497 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 14:16:01.402820 31210 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.406662 28317 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.405826 31497 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.415984 27506 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:01.418954 27516 catalog_manager.cc:5484] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:01.419592 27662 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.423839 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:16:01.424319 31462 tablet_replica.cc:330] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:16:01.424533 31462 raft_consensus.cc:2229] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:01.424794 31462 raft_consensus.cc:2258] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:01.426100 31462 ts_tablet_manager.cc:1875] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:01.435501 27924 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.436193 28186 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.437325 31462 ts_tablet_manager.cc:1888] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:01.437537 31462 log.cc:1196] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/89fb4ed8413f4eceb4348f9d216d3dd7
I20240503 14:16:01.438645 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet 89fb4ed8413f4eceb4348f9d216d3dd7 (table rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]) successfully deleted
I20240503 14:16:01.446759 31509 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20240503 14:16:01.448803 27793 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.453570 30304 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:01.680799 28317 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:16:01.683657 27659 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:01.683722 27924 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:01.683905 27793 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:01.684319 28449 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:01.684772 28055 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:01.684826 31513 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.685259 28186 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:01.685468 31513 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.686013 31512 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.686661 31511 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.687312 31118 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.689028 31118 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.691834 31513 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:01.694422 28055 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:01.694958 27922 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:01.697531 27659 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:01.698952 27793 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:01.701706 27512 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:01.706341 28186 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:01.708803 28317 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:01.709697 30304 raft_consensus.cc:1203] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:01.716399 30304 raft_consensus.cc:1203] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:01.718173 28057 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:16:01.718902 28256 consensus_peers.cc:485] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c23af4dc19c412685927c70cbee7124. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:01.719106 31512 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.720713 28187 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.722012 31512 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.723641 27924 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.724388 31512 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.730897 28449 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.731451 27793 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.731447 31512 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.731590 27659 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.732312 31118 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.732750 31118 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.733954 28451 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:01.735922 31118 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:01.738708 27924 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:01.739449 27793 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:01.744176 27659 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:01.745666 27507 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4c23af4dc19c412685927c70cbee7124 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:16:01.745030 27516 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:01.747193 28186 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:01.752360 28057 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:01.754355 28451 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:01.882916 31332 raft_consensus.cc:1050] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:16:01.883802 31332 consensus_queue.cc:237] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:01.887071 28057 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.887138 28317 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.887301 28186 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.888461 27924 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.889068 30035 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.888120 31332 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 14:16:01.890304 27793 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.890182 31562 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.890599 27662 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:01.891011 31509 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.891551 31332 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.892185 31497 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.894291 31121 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.895071 31210 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:01.907784 31210 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.911284 27791 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.915987 27924 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.917742 30035 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.918145 27521 catalog_manager.cc:5484] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.925616 28187 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.925791 27662 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.927856 28057 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.932255 28317 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
W20240503 14:16:01.935081 31121 consensus_queue.cc:1159] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7046cbd82d634976b53fe0cef5788f18" responder_term: 1 status { last_received { term: 1 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 1 index: 6 } }
W20240503 14:16:01.935719 31332 consensus_queue.cc:1159] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "078f8eff84c94dde9705ad1330922a11" responder_term: 1 status { last_received { term: 1 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 1 index: 6 } }
W20240503 14:16:01.936329 31562 consensus_queue.cc:1159] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" responder_term: 1 status { last_received { term: 1 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 1 index: 6 } }
I20240503 14:16:01.936976 28451 consensus_queue.cc:237] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:01.941192 27922 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.941246 27662 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.941347 28187 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.942037 28317 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.942222 27791 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.942274 30035 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:01.942371 31121 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.943017 31497 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.944474 31509 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.944780 31210 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.945312 31121 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.947311 31497 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:01.948848 31497 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.950127 27922 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.952078 27791 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.954638 28187 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.956163 27506 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d9c369355ad47f4843da19dc3ae3cac with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:01.957268 28319 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.962682 27520 catalog_manager.cc:5484] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.965775 27662 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.970489 30035 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:01.973269 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9c369355ad47f4843da19dc3ae3cac with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:16:01.983253 31571 tablet_replica.cc:330] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:16:01.984361 31571 raft_consensus.cc:2229] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:01.984653 31571 raft_consensus.cc:2258] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:01.985715 31571 ts_tablet_manager.cc:1875] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:01.991436 31571 ts_tablet_manager.cc:1888] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:01.991595 31571 log.cc:1196] T 0d9c369355ad47f4843da19dc3ae3cac P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/0d9c369355ad47f4843da19dc3ae3cac
I20240503 14:16:01.992416 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet 0d9c369355ad47f4843da19dc3ae3cac (table rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]) successfully deleted
I20240503 14:16:02.024156 28317 consensus_queue.cc:237] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } }
I20240503 14:16:02.027057 27662 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.027081 27791 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.027714 31511 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.028182 28187 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.028247 28057 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.028276 31511 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.028786 27922 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.029698 28451 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.030349 31535 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.030681 31512 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.030939 31513 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.032438 31118 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.036367 31513 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:02.036952 27793 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:02.037180 28057 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:02.038475 27922 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:02.042762 27662 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:02.043709 27521 catalog_manager.cc:5484] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:02.043619 28449 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:02.048154 28186 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } }
I20240503 14:16:02.058980 28317 consensus_queue.cc:237] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
W20240503 14:16:02.062597 28256 consensus_peers.cc:485] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05e093a194fc4500a9926c260ecbb308. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:02.062898 28057 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:02.062978 28186 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:02.063606 31513 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:02.064059 31535 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:02.064481 27922 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:02.064975 31535 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:02.066747 27793 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:02.067891 31513 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:02.069007 28451 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:02.069700 31513 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:02.073354 27922 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:02.074764 27793 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:02.076921 28449 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:02.070416 31513 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:02.082870 27516 catalog_manager.cc:5484] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:02.088835 27507 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 05e093a194fc4500a9926c260ecbb308 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:16:02.091399 28186 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:02.090926 28057 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:02.102605 27662 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:02.103575 31588 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:02.106415 27662 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:02.271657 31610 ts_tablet_manager.cc:902] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:16:02.273010 31610 tablet_copy_client.cc:308] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36073
I20240503 14:16:02.273844 28340 tablet_copy_service.cc:140] P acb93936f2f14e628eecb05d7d1f4d91: Received BeginTabletCopySession request for tablet 4c23af4dc19c412685927c70cbee7124 from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:54229)
I20240503 14:16:02.273988 28340 tablet_copy_service.cc:161] P acb93936f2f14e628eecb05d7d1f4d91: Beginning new tablet copy session on tablet 4c23af4dc19c412685927c70cbee7124 from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:54229: session id = e938d875af584f1f88a2d4b50c589268-4c23af4dc19c412685927c70cbee7124
I20240503 14:16:02.276168 28340 tablet_copy_source_session.cc:205] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:02.277523 31610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23af4dc19c412685927c70cbee7124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.282508 31610 tablet_copy_client.cc:783] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:02.282900 31610 tablet_copy_client.cc:647] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:02.284875 31368 raft_consensus.cc:1050] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:16:02.284987 31610 tablet_copy_client.cc:523] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:02.286020 31368 consensus_queue.cc:237] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:02.287823 31610 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:16:02.289892 27922 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.290463 30304 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.290946 31615 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.291599 31615 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.303270 27662 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.303270 27793 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.303910 28449 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.303963 31615 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.304744 31615 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.305322 31615 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.318181 31432 raft_consensus.cc:1050] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:16:02.320117 31432 consensus_queue.cc:237] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:02.321866 28317 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.321442 31617 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.325450 31615 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.329964 27513 catalog_manager.cc:5484] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:02.330273 28449 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.334209 31610 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:02.334862 31610 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:16:02.335194 31610 ts_tablet_manager.cc:1367] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.047s user 0.046s sys 0.001s
I20240503 14:16:02.336174 31610 raft_consensus.cc:348] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:02.336580 31610 raft_consensus.cc:729] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:16:02.336822 31610 consensus_queue.cc:260] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:02.337543 31610 ts_tablet_manager.cc:1398] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:02.338923 28340 tablet_copy_service.cc:342] P acb93936f2f14e628eecb05d7d1f4d91: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-4c23af4dc19c412685927c70cbee7124 received from {username='slave'} at 127.10.145.8:54229
I20240503 14:16:02.339098 28340 tablet_copy_service.cc:433] P acb93936f2f14e628eecb05d7d1f4d91: ending tablet copy session e938d875af584f1f88a2d4b50c589268-4c23af4dc19c412685927c70cbee7124 on tablet 4c23af4dc19c412685927c70cbee7124 with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:16:02.341931 30304 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.342854 27922 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.347826 28057 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.351104 27791 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.353406 28317 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.357606 31615 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.361691 30304 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.362495 31625 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.364853 27922 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.365654 27793 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.366200 27662 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.370265 31625 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.371023 31625 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.379915 31635 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.390203 28057 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.396935 27659 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.416054 28317 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.415427 31632 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.416659 28449 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.416785 31634 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.416996 28186 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:02.417474 31625 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.420132 28449 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.422616 31634 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:02.423748 27922 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.424329 27791 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.424978 28186 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.430408 27513 catalog_manager.cc:5484] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.440491 28317 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.440779 27662 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.450250 31610 ts_tablet_manager.cc:902] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:16:02.452625 31610 tablet_copy_client.cc:308] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36073
I20240503 14:16:02.453395 28340 tablet_copy_service.cc:140] P acb93936f2f14e628eecb05d7d1f4d91: Received BeginTabletCopySession request for tablet 05e093a194fc4500a9926c260ecbb308 from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:54229)
I20240503 14:16:02.453625 28340 tablet_copy_service.cc:161] P acb93936f2f14e628eecb05d7d1f4d91: Beginning new tablet copy session on tablet 05e093a194fc4500a9926c260ecbb308 from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:54229: session id = e938d875af584f1f88a2d4b50c589268-05e093a194fc4500a9926c260ecbb308
I20240503 14:16:02.455897 30303 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.456482 28340 tablet_copy_source_session.cc:205] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:02.458628 31610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e093a194fc4500a9926c260ecbb308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.459599 28057 consensus_queue.cc:237] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:02.464200 27662 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.466468 31610 tablet_copy_client.cc:783] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:02.466814 31610 tablet_copy_client.cc:647] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:02.466825 27791 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.466971 28451 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.467871 28186 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.468809 30304 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.468806 31625 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.469530 31625 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.469962 31625 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.470373 31625 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.470777 31625 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.474301 27922 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.476291 31634 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.477260 31610 tablet_copy_client.cc:523] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:02.478292 27662 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.478771 31625 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.481882 28451 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.483296 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e9e6daafc14437faff996cfb40f6dfb with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:02.484843 27791 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.485440 28186 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.486323 27516 catalog_manager.cc:5484] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:02.487346 30304 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.490562 27922 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.496054 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9e6daafc14437faff996cfb40f6dfb with delete_type TABLET_DATA_TOMBSTONED (TS acb93936f2f14e628eecb05d7d1f4d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39840
I20240503 14:16:02.497413 31657 tablet_replica.cc:330] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: stopping tablet replica
I20240503 14:16:02.497746 31657 raft_consensus.cc:2229] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:02.498128 31657 raft_consensus.cc:2258] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:02.500753 31657 ts_tablet_manager.cc:1875] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:02.509395 31657 ts_tablet_manager.cc:1888] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:02.509660 31657 log.cc:1196] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/0e9e6daafc14437faff996cfb40f6dfb
I20240503 14:16:02.510594 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet 0e9e6daafc14437faff996cfb40f6dfb (table rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]) successfully deleted
I20240503 14:16:02.517021 31610 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:16:02.567139 31610 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:02.567586 31610 tablet_bootstrap.cc:492] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:16:02.567871 31610 ts_tablet_manager.cc:1367] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.051s user 0.040s sys 0.007s
I20240503 14:16:02.568838 31610 raft_consensus.cc:348] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:02.569103 31610 raft_consensus.cc:729] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:16:02.569345 31610 consensus_queue.cc:260] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:02.570025 31610 ts_tablet_manager.cc:1398] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:02.571007 28340 tablet_copy_service.cc:342] P acb93936f2f14e628eecb05d7d1f4d91: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-05e093a194fc4500a9926c260ecbb308 received from {username='slave'} at 127.10.145.8:54229
I20240503 14:16:02.571130 28340 tablet_copy_service.cc:433] P acb93936f2f14e628eecb05d7d1f4d91: ending tablet copy session e938d875af584f1f88a2d4b50c589268-05e093a194fc4500a9926c260ecbb308 on tablet 05e093a194fc4500a9926c260ecbb308 with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:16:02.807354 30035 raft_consensus.cc:1203] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:02.963956 28186 consensus_queue.cc:237] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:02.966873 27791 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.966883 27659 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.966902 30304 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.966996 28057 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.967391 27922 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.967512 31685 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.967797 31626 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.968350 31622 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.968839 31622 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.969089 31685 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.970626 28451 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:02.971926 31685 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:02.972380 31622 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.972735 28057 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.972797 27922 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.975569 28451 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.976466 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 29945558359248c6a8a4c68952cfcc81 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:02.976979 27791 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.981853 30304 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.982739 27659 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.980891 27513 catalog_manager.cc:5484] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 6 to 7, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:02.988612 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet 29945558359248c6a8a4c68952cfcc81 with delete_type TABLET_DATA_TOMBSTONED (TS acb93936f2f14e628eecb05d7d1f4d91 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39840
I20240503 14:16:02.988989 31657 tablet_replica.cc:330] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91: stopping tablet replica
I20240503 14:16:02.989172 31657 raft_consensus.cc:2229] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:02.989454 31657 raft_consensus.cc:2258] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:02.990319 31657 ts_tablet_manager.cc:1875] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:02.995774 31657 ts_tablet_manager.cc:1888] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:02.995877 31657 log.cc:1196] T 29945558359248c6a8a4c68952cfcc81 P acb93936f2f14e628eecb05d7d1f4d91: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/29945558359248c6a8a4c68952cfcc81
I20240503 14:16:02.997344 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet 29945558359248c6a8a4c68952cfcc81 (table rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]) successfully deleted
I20240503 14:16:03.070173 30035 raft_consensus.cc:1203] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:03.265563 31588 raft_consensus.cc:1050] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:16:03.266490 31588 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:03.269452 27791 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.269556 27922 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.270007 31512 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.270256 28186 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.270380 28451 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.270620 30035 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.270903 28057 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.270990 27659 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.271353 31513 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.271632 31701 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.272017 31512 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.272230 31513 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.272747 31714 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.273797 31588 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.287997 31701 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.288571 30035 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.289217 27662 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.293679 27791 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.294005 28186 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.294458 28057 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.298182 27516 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 reported cstate change: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.304047 28451 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.304098 27922 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.309293 28317 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:03.313112 27922 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.313150 27791 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.313369 28186 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.313730 31588 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.313808 30035 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.313992 31513 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.314199 28451 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.314400 27659 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.314697 31513 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.314970 31715 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.315518 31701 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.316149 31714 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.317538 31588 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.319183 27922 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.319272 30035 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.319276 28451 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.322384 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c23af4dc19c412685927c70cbee7124 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:03.323210 28186 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.323942 27659 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.324939 27512 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.326793 27791 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.331877 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c23af4dc19c412685927c70cbee7124 with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:16:03.335873 31723 tablet_replica.cc:330] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:16:03.336084 31723 raft_consensus.cc:2229] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:03.337106 31723 raft_consensus.cc:2258] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:03.338142 31723 ts_tablet_manager.cc:1875] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:03.342674 31723 ts_tablet_manager.cc:1888] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:03.342793 31723 log.cc:1196] T 4c23af4dc19c412685927c70cbee7124 P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c23af4dc19c412685927c70cbee7124
I20240503 14:16:03.343472 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet 4c23af4dc19c412685927c70cbee7124 (table rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]) successfully deleted
I20240503 14:16:03.448693 31512 raft_consensus.cc:1050] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:16:03.449791 31512 consensus_queue.cc:237] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:03.452807 28451 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.452807 27659 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.453261 27922 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.453778 27791 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.454077 31701 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.454624 31701 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.455404 31513 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.455777 31701 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.457917 30035 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.458523 28186 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.458508 31714 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.459450 31513 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.462976 28057 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:03.463487 31701 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:03.484879 31714 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.491935 28188 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.492645 27922 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.496109 28057 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.497865 27516 catalog_manager.cc:5484] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.499045 28451 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.499310 27659 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.508692 27791 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.515965 30035 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.522353 28317 consensus_queue.cc:237] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:03.524820 27659 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.525202 27922 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.525808 31513 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.526171 27793 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.526307 31513 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.527382 30035 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.529609 31512 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.530081 31512 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.532495 28186 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.533613 31714 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.533607 31513 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.535818 27791 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.539404 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 05e093a194fc4500a9926c260ecbb308 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:03.539451 28186 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.540434 27922 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.541977 30035 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.542253 27659 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.543473 27520 catalog_manager.cc:5484] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:03.547570 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e093a194fc4500a9926c260ecbb308 with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240503 14:16:03.548002 31723 tablet_replica.cc:330] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:16:03.548193 31723 raft_consensus.cc:2229] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:03.548547 31723 raft_consensus.cc:2258] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:03.549875 31723 ts_tablet_manager.cc:1875] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:03.559805 28451 raft_consensus.cc:1261] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:03.560652 31513 consensus_queue.cc:1035] T 05e093a194fc4500a9926c260ecbb308 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:03.562919 28451 raft_consensus.cc:2932] T 05e093a194fc4500a9926c260ecbb308 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:03.565937 31723 ts_tablet_manager.cc:1888] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:03.566385 31723 log.cc:1196] T 05e093a194fc4500a9926c260ecbb308 P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/05e093a194fc4500a9926c260ecbb308
I20240503 14:16:03.567162 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet 05e093a194fc4500a9926c260ecbb308 (table rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]) successfully deleted
I20240503 14:16:04.113037 30149 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.119621 28035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:04.129839 27642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:04.131724 30284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.159003 28430 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:04.160881 27772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:04.223829 27904 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:04.238830 28299 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:04.294921 28168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:04.306185 30016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.487818 30284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.512434 27642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:04.514261 28035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:04.524417 27772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:04.525604 28430 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:04.547147 30149 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.562090 27904 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:04.588758 28168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:04.596977 28299 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:04.671240 30016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.768872 28057 consensus_queue.cc:237] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:04.772141 30304 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.772317 27924 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:16:04.773526 27993 consensus_peers.cc:485] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e9e6daafc14437faff996cfb40f6dfb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:04.773612 31791 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.774075 31635 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.778652 27791 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.779075 28451 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.779194 27659 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.779737 31635 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.780297 28186 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.780396 31842 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.780745 31791 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.781463 31635 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.795166 31842 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:04.798004 27791 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:04.799199 27662 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:04.806659 27520 catalog_manager.cc:5484] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:04.809412 27924 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:04.809829 30304 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:04.809841 28186 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:04.817185 28451 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:04.839082 27922 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:04.842604 28451 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.843595 31254 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.844532 28186 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.844583 28057 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.845641 31861 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.846269 27659 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.846267 31861 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.847043 31864 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.853513 30169 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.854566 27791 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.855628 31254 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.856724 31254 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:04.861068 27862 consensus_peers.cc:485] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e895331729d3400384f20aa3895540de. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:04.874363 31254 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:04.877804 27662 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:04.878086 28055 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:04.882532 27791 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:04.886869 27521 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:04.889014 28188 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:04.889703 30169 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:04.890445 28451 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:04.912616 27662 consensus_queue.cc:237] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:04.915995 30304 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.916669 31790 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.918516 28451 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.918835 27922 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.918993 27791 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.919698 31669 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.920231 31790 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.920796 31790 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.927940 28055 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:16:04.928314 27598 consensus_peers.cc:485] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77469d6a6610402ca96598bb695224d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:04.928901 31790 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.929767 28188 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.931159 31669 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.937429 31790 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:04.938203 27922 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:04.943888 27521 catalog_manager.cc:5484] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:04.950103 30303 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:04.950577 28451 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:04.949301 27791 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:04.952126 28055 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:04.969044 28186 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:04.981156 28186 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
W20240503 14:16:04.985008 28124 consensus_peers.cc:485] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12dae8a392374b91be378ec431838686. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:04.984997 27922 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.985669 31742 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.991067 27662 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.991520 28055 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.991541 30169 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.991736 28317 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.992372 27791 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:04.992383 31742 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.992957 31893 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.994542 31895 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.995479 31742 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.996143 31896 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:04.997200 31742 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:04.998054 27791 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.005600 27521 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.008234 30169 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.008352 27922 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.020438 27662 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.025498 28317 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.033937 28451 consensus_queue.cc:237] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:05.033700 28055 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.037098 30035 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.037099 28186 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.038070 31702 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.038632 28317 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.038719 31702 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.039664 31702 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:05.060129 28387 consensus_peers.cc:485] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d9c369355ad47f4843da19dc3ae3cac. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.061034 27662 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.060995 27791 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.061547 31210 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.065774 31702 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.067890 31914 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.070367 28186 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.070888 28317 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.072449 27518 catalog_manager.cc:5484] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:05.073809 27922 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.077172 27793 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.078379 31916 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.083599 27922 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.081248 30035 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.095607 27662 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.142930 27662 consensus_queue.cc:237] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:05.159466 30303 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.159819 27793 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.159907 28450 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.160336 27922 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.160955 31669 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.161465 28186 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.161532 31790 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.162019 28055 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.162124 31790 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.162549 31879 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.162838 31669 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.163290 31790 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:05.163717 27598 consensus_peers.cc:485] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56dad6fc0ba0440aa4660030b1bb9267. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.181612 31879 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:05.187544 28450 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:05.189626 27521 catalog_manager.cc:5484] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:05.194422 31928 ts_tablet_manager.cc:902] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:16:05.206666 28186 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:05.206785 27793 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:05.208999 31928 tablet_copy_client.cc:308] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:45951
I20240503 14:16:05.209842 28077 tablet_copy_service.cc:140] P 078f8eff84c94dde9705ad1330922a11: Received BeginTabletCopySession request for tablet 0e9e6daafc14437faff996cfb40f6dfb from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:57647)
I20240503 14:16:05.210052 28077 tablet_copy_service.cc:161] P 078f8eff84c94dde9705ad1330922a11: Beginning new tablet copy session on tablet 0e9e6daafc14437faff996cfb40f6dfb from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:57647: session id = 54dab61f1a204c0da4f56f98b3fd2690-0e9e6daafc14437faff996cfb40f6dfb
I20240503 14:16:05.211091 28055 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:05.213070 28077 tablet_copy_source_session.cc:205] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.216071 30303 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:05.217273 27922 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:05.219627 31928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9e6daafc14437faff996cfb40f6dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.228073 31928 tablet_copy_client.cc:783] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.228439 31928 tablet_copy_client.cc:647] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.230612 31928 tablet_copy_client.cc:523] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.234983 31928 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:16:05.242070 28317 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:05.246021 30035 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.248452 27922 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.248929 31701 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.252696 31701 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:05.258996 28256 consensus_peers.cc:485] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c23af4dc19c412685927c70cbee7124. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.276203 28450 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.276393 27793 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.276639 27662 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.276655 28186 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.277326 31948 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.277870 31714 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.278122 31701 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.278666 31701 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.303948 31701 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.304597 27662 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.310377 27791 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.311969 28186 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.312175 30035 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.314169 28450 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.320348 27518 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 reported cstate change: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.325254 31928 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.325740 31928 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:16:05.326037 31928 ts_tablet_manager.cc:1367] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.091s user 0.047s sys 0.006s
I20240503 14:16:05.326941 31928 raft_consensus.cc:348] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:05.327298 31928 raft_consensus.cc:729] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:16:05.329485 31928 consensus_queue.cc:260] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:05.331369 31928 ts_tablet_manager.cc:1398] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:16:05.332147 28077 tablet_copy_service.cc:342] P 078f8eff84c94dde9705ad1330922a11: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-0e9e6daafc14437faff996cfb40f6dfb received from {username='slave'} at 127.10.145.9:57647
I20240503 14:16:05.332288 28077 tablet_copy_service.cc:433] P 078f8eff84c94dde9705ad1330922a11: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-0e9e6daafc14437faff996cfb40f6dfb on tablet 0e9e6daafc14437faff996cfb40f6dfb with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:16:05.339758 27922 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.350852 31963 ts_tablet_manager.cc:902] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:16:05.352592 31963 tablet_copy_client.cc:308] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:36119
I20240503 14:16:05.360818 27946 tablet_copy_service.cc:140] P a242f99f54b4494893064cfe3030b4ad: Received BeginTabletCopySession request for tablet e895331729d3400384f20aa3895540de from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:46423)
I20240503 14:16:05.361030 27946 tablet_copy_service.cc:161] P a242f99f54b4494893064cfe3030b4ad: Beginning new tablet copy session on tablet e895331729d3400384f20aa3895540de from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:46423: session id = 01173ed91fd341e3a4a4599118167e52-e895331729d3400384f20aa3895540de
I20240503 14:16:05.362618 28450 consensus_queue.cc:237] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:05.363380 27946 tablet_copy_source_session.cc:205] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.365077 31963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e895331729d3400384f20aa3895540de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.366125 27922 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.366523 27791 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.367372 28317 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.367617 27662 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.368184 31914 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.369437 31914 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:05.370592 28387 consensus_peers.cc:485] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89fb4ed8413f4eceb4348f9d216d3dd7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.370898 31915 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.371232 28186 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.371503 31915 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.372318 31963 tablet_copy_client.cc:783] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.372611 31963 tablet_copy_client.cc:647] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.375356 30303 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.376091 31963 tablet_copy_client.cc:523] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.376720 31915 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.377396 31915 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.384438 31915 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.386690 28317 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.388739 31963 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:16:05.389753 30303 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.392849 27521 catalog_manager.cc:5484] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.394373 27791 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.400084 27922 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.404482 28186 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.413690 27662 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.460173 31986 ts_tablet_manager.cc:902] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:16:05.463881 28055 consensus_queue.cc:237] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:05.469499 31986 tablet_copy_client.cc:308] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:35261
I20240503 14:16:05.470436 27682 tablet_copy_service.cc:140] P 7046cbd82d634976b53fe0cef5788f18: Received BeginTabletCopySession request for tablet 77469d6a6610402ca96598bb695224d9 from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:47681)
I20240503 14:16:05.470690 27682 tablet_copy_service.cc:161] P 7046cbd82d634976b53fe0cef5788f18: Beginning new tablet copy session on tablet 77469d6a6610402ca96598bb695224d9 from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:47681: session id = e938d875af584f1f88a2d4b50c589268-77469d6a6610402ca96598bb695224d9
I20240503 14:16:05.475908 27682 tablet_copy_source_session.cc:205] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.478101 31986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77469d6a6610402ca96598bb695224d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.481990 27922 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.482082 30303 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.483348 31849 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.484176 31849 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.484584 31963 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.485085 31963 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:16:05.485467 31963 ts_tablet_manager.cc:1367] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.097s user 0.044s sys 0.014s
W20240503 14:16:05.486946 27993 consensus_peers.cc:485] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b178a32f4b24249aeae16cada4a3d56. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.488279 31963 raft_consensus.cc:348] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:05.488893 31963 raft_consensus.cc:729] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:16:05.489168 31963 consensus_queue.cc:260] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:05.490012 31963 ts_tablet_manager.cc:1398] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:16:05.492839 27946 tablet_copy_service.cc:342] P a242f99f54b4494893064cfe3030b4ad: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-e895331729d3400384f20aa3895540de received from {username='slave'} at 127.10.145.10:46423
I20240503 14:16:05.493016 27946 tablet_copy_service.cc:433] P a242f99f54b4494893064cfe3030b4ad: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-e895331729d3400384f20aa3895540de on tablet e895331729d3400384f20aa3895540de with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:16:05.494947 27662 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.496889 27793 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.497640 31791 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.498584 31791 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.502749 31963 ts_tablet_manager.cc:902] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605)
I20240503 14:16:05.514037 31986 tablet_copy_client.cc:783] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.514442 31986 tablet_copy_client.cc:647] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.514963 28450 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.516234 31849 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.517377 31986 tablet_copy_client.cc:523] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.518322 28186 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.521498 30303 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.527525 31849 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.529992 31849 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.532150 31986 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:16:05.534937 27791 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.545675 27515 catalog_manager.cc:5484] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 reported cstate change: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.548895 27922 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.554335 28186 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.562112 28451 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.563468 27659 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.574199 31963 tablet_copy_client.cc:308] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.5:35605
I20240503 14:16:05.575069 28208 tablet_copy_service.cc:140] P c91f285b8f4f4a27bcdd0ada303c1422: Received BeginTabletCopySession request for tablet 12dae8a392374b91be378ec431838686 from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:37207)
I20240503 14:16:05.575259 28208 tablet_copy_service.cc:161] P c91f285b8f4f4a27bcdd0ada303c1422: Beginning new tablet copy session on tablet 12dae8a392374b91be378ec431838686 from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:37207: session id = 01173ed91fd341e3a4a4599118167e52-12dae8a392374b91be378ec431838686
I20240503 14:16:05.579802 28208 tablet_copy_source_session.cc:205] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.584544 31963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dae8a392374b91be378ec431838686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.593875 31928 ts_tablet_manager.cc:902] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:16:05.593928 31963 tablet_copy_client.cc:783] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.594215 31963 tablet_copy_client.cc:647] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.595769 32009 ts_tablet_manager.cc:902] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:16:05.597142 31963 tablet_copy_client.cc:523] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.625667 30168 raft_consensus.cc:1203] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:05.629029 31995 ts_tablet_manager.cc:902] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:16:05.632552 31928 tablet_copy_client.cc:308] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.7:43675
I20240503 14:16:05.633741 28471 tablet_copy_service.cc:140] P 9231e715a4e342a197d6c5456ba6b399: Received BeginTabletCopySession request for tablet 0d9c369355ad47f4843da19dc3ae3cac from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:54935)
I20240503 14:16:05.633926 28471 tablet_copy_service.cc:161] P 9231e715a4e342a197d6c5456ba6b399: Beginning new tablet copy session on tablet 0d9c369355ad47f4843da19dc3ae3cac from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:54935: session id = 54dab61f1a204c0da4f56f98b3fd2690-0d9c369355ad47f4843da19dc3ae3cac
I20240503 14:16:05.637029 28471 tablet_copy_source_session.cc:205] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.639495 31928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9c369355ad47f4843da19dc3ae3cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.641968 31963 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:16:05.657001 31995 tablet_copy_client.cc:308] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36073
I20240503 14:16:05.661783 28055 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:05.662591 31928 tablet_copy_client.cc:783] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.663307 31928 tablet_copy_client.cc:647] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.668460 31928 tablet_copy_client.cc:523] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.669116 28340 tablet_copy_service.cc:140] P acb93936f2f14e628eecb05d7d1f4d91: Received BeginTabletCopySession request for tablet 4c23af4dc19c412685927c70cbee7124 from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:33777)
I20240503 14:16:05.669327 28340 tablet_copy_service.cc:161] P acb93936f2f14e628eecb05d7d1f4d91: Beginning new tablet copy session on tablet 4c23af4dc19c412685927c70cbee7124 from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:33777: session id = 01173ed91fd341e3a4a4599118167e52-4c23af4dc19c412685927c70cbee7124
I20240503 14:16:05.671677 28340 tablet_copy_source_session.cc:205] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.672672 31928 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:16:05.673581 31995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23af4dc19c412685927c70cbee7124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.680687 28188 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:16:05.681592 27993 consensus_peers.cc:485] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f00380931064f71bc78626af72488f3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.681759 28451 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.682380 32024 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.682775 31791 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.684054 31995 tablet_copy_client.cc:783] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.684366 31995 tablet_copy_client.cc:647] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.684424 27922 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.685117 27791 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.685351 30168 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.685842 31849 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.686515 31995 tablet_copy_client.cc:523] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.686488 31849 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.688645 27659 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.689457 32026 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.695125 31791 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.701448 31986 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.701828 31995 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:16:05.704589 31986 tablet_bootstrap.cc:492] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:16:05.705839 31986 ts_tablet_manager.cc:1367] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.174s user 0.051s sys 0.005s
I20240503 14:16:05.708230 31986 raft_consensus.cc:348] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:05.710925 31986 raft_consensus.cc:729] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:16:05.711206 31986 consensus_queue.cc:260] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:05.712992 31986 ts_tablet_manager.cc:1398] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:16:05.713927 27682 tablet_copy_service.cc:342] P 7046cbd82d634976b53fe0cef5788f18: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-77469d6a6610402ca96598bb695224d9 received from {username='slave'} at 127.10.145.8:47681
I20240503 14:16:05.714089 27682 tablet_copy_service.cc:433] P 7046cbd82d634976b53fe0cef5788f18: ending tablet copy session e938d875af584f1f88a2d4b50c589268-77469d6a6610402ca96598bb695224d9 on tablet 77469d6a6610402ca96598bb695224d9 with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:16:05.744274 30303 raft_consensus.cc:1203] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:05.751544 32009 tablet_copy_client.cc:308] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:35261
I20240503 14:16:05.752442 27682 tablet_copy_service.cc:140] P 7046cbd82d634976b53fe0cef5788f18: Received BeginTabletCopySession request for tablet 56dad6fc0ba0440aa4660030b1bb9267 from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:47681)
I20240503 14:16:05.752661 27682 tablet_copy_service.cc:161] P 7046cbd82d634976b53fe0cef5788f18: Beginning new tablet copy session on tablet 56dad6fc0ba0440aa4660030b1bb9267 from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:47681: session id = e938d875af584f1f88a2d4b50c589268-56dad6fc0ba0440aa4660030b1bb9267
I20240503 14:16:05.755278 31791 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.756453 27682 tablet_copy_source_session.cc:205] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.758229 28188 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.763540 27791 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.764288 27659 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.767261 30169 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.771098 27515 catalog_manager.cc:5484] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:05.776000 27922 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.783706 32009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56dad6fc0ba0440aa4660030b1bb9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.791400 28451 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:05.797756 32009 tablet_copy_client.cc:783] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.798045 32009 tablet_copy_client.cc:647] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.801589 32009 tablet_copy_client.cc:523] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.801959 32036 ts_tablet_manager.cc:902] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
I20240503 14:16:05.805480 32009 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:16:05.825562 32036 tablet_copy_client.cc:308] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.7:43675
I20240503 14:16:05.827323 28471 tablet_copy_service.cc:140] P 9231e715a4e342a197d6c5456ba6b399: Received BeginTabletCopySession request for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:54935)
I20240503 14:16:05.827562 28471 tablet_copy_service.cc:161] P 9231e715a4e342a197d6c5456ba6b399: Beginning new tablet copy session on tablet 89fb4ed8413f4eceb4348f9d216d3dd7 from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:54935: session id = 54dab61f1a204c0da4f56f98b3fd2690-89fb4ed8413f4eceb4348f9d216d3dd7
I20240503 14:16:05.831790 28471 tablet_copy_source_session.cc:205] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.835690 32036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fb4ed8413f4eceb4348f9d216d3dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.841571 31995 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.842084 31995 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:16:05.842424 31995 ts_tablet_manager.cc:1367] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.141s user 0.046s sys 0.009s
I20240503 14:16:05.843283 31995 raft_consensus.cc:348] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:05.843636 31995 raft_consensus.cc:729] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:16:05.843856 31995 consensus_queue.cc:260] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:05.844336 32036 tablet_copy_client.cc:783] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.844604 32036 tablet_copy_client.cc:647] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.847198 31995 ts_tablet_manager.cc:1398] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:16:05.850096 32036 tablet_copy_client.cc:523] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.851589 28340 tablet_copy_service.cc:342] P acb93936f2f14e628eecb05d7d1f4d91: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-4c23af4dc19c412685927c70cbee7124 received from {username='slave'} at 127.10.145.10:33777
I20240503 14:16:05.851765 28340 tablet_copy_service.cc:433] P acb93936f2f14e628eecb05d7d1f4d91: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-4c23af4dc19c412685927c70cbee7124 on tablet 4c23af4dc19c412685927c70cbee7124 with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:16:05.854036 32036 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:16:05.864806 28186 consensus_queue.cc:237] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:05.887097 31963 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.887413 32009 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.887856 31963 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:16:05.887881 32009 tablet_bootstrap.cc:492] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:16:05.888227 31963 ts_tablet_manager.cc:1367] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.249s user 0.058s sys 0.005s
I20240503 14:16:05.889202 31963 raft_consensus.cc:348] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:05.889709 31963 raft_consensus.cc:729] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:16:05.889998 31963 consensus_queue.cc:260] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:05.890766 31963 ts_tablet_manager.cc:1398] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:05.891391 32009 ts_tablet_manager.cc:1367] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.086s user 0.053s sys 0.009s
I20240503 14:16:05.892412 30035 raft_consensus.cc:1203] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:05.892328 32009 raft_consensus.cc:348] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:05.892679 32009 raft_consensus.cc:729] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:16:05.893004 28208 tablet_copy_service.cc:342] P c91f285b8f4f4a27bcdd0ada303c1422: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-12dae8a392374b91be378ec431838686 received from {username='slave'} at 127.10.145.10:37207
I20240503 14:16:05.893178 28208 tablet_copy_service.cc:433] P c91f285b8f4f4a27bcdd0ada303c1422: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-12dae8a392374b91be378ec431838686 on tablet 12dae8a392374b91be378ec431838686 with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:16:05.892921 32009 consensus_queue.cc:260] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:05.894577 32009 ts_tablet_manager.cc:1398] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:16:05.896788 27682 tablet_copy_service.cc:342] P 7046cbd82d634976b53fe0cef5788f18: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-56dad6fc0ba0440aa4660030b1bb9267 received from {username='slave'} at 127.10.145.8:47681
I20240503 14:16:05.896939 27682 tablet_copy_service.cc:433] P 7046cbd82d634976b53fe0cef5788f18: ending tablet copy session e938d875af584f1f88a2d4b50c589268-56dad6fc0ba0440aa4660030b1bb9267 on tablet 56dad6fc0ba0440aa4660030b1bb9267 with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:16:05.905845 31928 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.907801 27922 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.907801 30303 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:16:05.909689 28124 consensus_peers.cc:485] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29945558359248c6a8a4c68952cfcc81. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.910096 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.911370 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.921648 31928 tablet_bootstrap.cc:492] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:16:05.922024 31928 ts_tablet_manager.cc:1367] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.249s user 0.042s sys 0.015s
I20240503 14:16:05.923172 31928 raft_consensus.cc:348] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:05.923527 31928 raft_consensus.cc:729] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:16:05.923791 31928 consensus_queue.cc:260] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:05.924564 31928 ts_tablet_manager.cc:1398] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:16:05.927963 32056 ts_tablet_manager.cc:902] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:16:05.928066 28471 tablet_copy_service.cc:342] P 9231e715a4e342a197d6c5456ba6b399: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-0d9c369355ad47f4843da19dc3ae3cac received from {username='slave'} at 127.10.145.9:54935
I20240503 14:16:05.928203 28471 tablet_copy_service.cc:433] P 9231e715a4e342a197d6c5456ba6b399: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-0d9c369355ad47f4843da19dc3ae3cac on tablet 0d9c369355ad47f4843da19dc3ae3cac with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:16:05.934433 27791 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.934571 27659 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.935277 32036 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.935205 32053 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.935562 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.935768 32036 tablet_bootstrap.cc:492] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:16:05.936067 32036 ts_tablet_manager.cc:1367] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.082s user 0.051s sys 0.000s
I20240503 14:16:05.938203 28451 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.938935 32053 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.939364 32036 raft_consensus.cc:348] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:05.939738 32036 raft_consensus.cc:729] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:16:05.940075 32036 consensus_queue.cc:260] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:05.939952 32053 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.941601 32036 ts_tablet_manager.cc:1398] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:16:05.941442 27659 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.942346 28471 tablet_copy_service.cc:342] P 9231e715a4e342a197d6c5456ba6b399: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-89fb4ed8413f4eceb4348f9d216d3dd7 received from {username='slave'} at 127.10.145.9:54935
I20240503 14:16:05.942497 28471 tablet_copy_service.cc:433] P 9231e715a4e342a197d6c5456ba6b399: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-89fb4ed8413f4eceb4348f9d216d3dd7 on tablet 89fb4ed8413f4eceb4348f9d216d3dd7 with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:16:05.957736 27521 catalog_manager.cc:5484] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.961511 30303 raft_consensus.cc:1203] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:05.968683 27791 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.972337 28450 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.979066 28055 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:05.980217 32049 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:05.981098 27922 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.986086 30303 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:05.991598 30168 raft_consensus.cc:1203] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:05.994411 28055 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:06.008190 32056 tablet_copy_client.cc:308] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:45951
I20240503 14:16:06.008967 31992 raft_consensus.cc:1050] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:16:06.009093 28077 tablet_copy_service.cc:140] P 078f8eff84c94dde9705ad1330922a11: Received BeginTabletCopySession request for tablet 6b178a32f4b24249aeae16cada4a3d56 from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:57647)
I20240503 14:16:06.009305 28077 tablet_copy_service.cc:161] P 078f8eff84c94dde9705ad1330922a11: Beginning new tablet copy session on tablet 6b178a32f4b24249aeae16cada4a3d56 from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:57647: session id = 54dab61f1a204c0da4f56f98b3fd2690-6b178a32f4b24249aeae16cada4a3d56
I20240503 14:16:06.010413 31992 consensus_queue.cc:237] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:06.012792 28077 tablet_copy_source_session.cc:205] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:06.016638 32056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b178a32f4b24249aeae16cada4a3d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.024492 32056 tablet_copy_client.cc:783] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:06.024971 32056 tablet_copy_client.cc:647] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:06.027212 32056 tablet_copy_client.cc:523] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:06.028995 27922 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.029101 30303 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.029603 27659 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.029939 30168 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.030233 27791 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.030405 31992 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.031157 28450 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.031556 28186 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.031569 31849 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.031895 32024 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.032173 31791 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.032516 32026 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.032770 31791 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.033074 30035 raft_consensus.cc:1203] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:06.043912 32056 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:16:06.044925 28317 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:06.054337 31791 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:16:06.072499 28256 consensus_peers.cc:485] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: add941da26824cdda7489b5da990e879. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:06.075031 27924 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:06.075784 30169 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:06.076273 27659 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:06.076783 27791 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:06.077311 31948 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:06.078704 31958 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:06.079422 31958 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:06.079787 31948 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:06.085124 28451 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:06.085979 31948 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:06.098367 28186 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:06.104378 31958 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:06.106348 32085 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:06.118304 27659 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:06.127586 27515 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:06.128031 28451 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:06.140506 30304 raft_consensus.cc:1203] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:06.140483 30169 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:06.157452 31791 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.161839 28449 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.171659 27521 catalog_manager.cc:5484] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:06.170015 27659 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.176220 28186 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.176736 27793 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.190302 30169 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.201611 28186 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:06.212129 30303 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.220707 31963 ts_tablet_manager.cc:902] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:16:06.224779 27924 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:06.239912 27791 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:06.247880 28055 consensus_queue.cc:237] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:06.260946 32056 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:06.261085 28451 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.261523 32056 tablet_bootstrap.cc:492] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:16:06.261844 32056 ts_tablet_manager.cc:1367] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.218s user 0.084s sys 0.047s
I20240503 14:16:06.261847 32075 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.262692 32056 raft_consensus.cc:348] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:06.263012 32056 raft_consensus.cc:729] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:16:06.263235 32056 consensus_queue.cc:260] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:06.263875 32056 ts_tablet_manager.cc:1398] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:16:06.264560 28077 tablet_copy_service.cc:342] P 078f8eff84c94dde9705ad1330922a11: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-6b178a32f4b24249aeae16cada4a3d56 received from {username='slave'} at 127.10.145.9:57647
I20240503 14:16:06.264722 28077 tablet_copy_service.cc:433] P 078f8eff84c94dde9705ad1330922a11: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-6b178a32f4b24249aeae16cada4a3d56 on tablet 6b178a32f4b24249aeae16cada4a3d56 with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:16:06.266378 30169 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.266906 28186 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.265003 27922 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.267092 27791 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.268288 32075 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.269527 32075 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.269762 31791 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.271131 30303 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.276000 32075 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.281383 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e9e6daafc14437faff996cfb40f6dfb with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:06.282799 32075 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.283033 27512 catalog_manager.cc:5484] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:06.284191 30169 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.288762 31871 raft_consensus.cc:1050] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:16:06.290311 31871 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:06.292214 28186 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.294474 27642 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9e6daafc14437faff996cfb40f6dfb with delete_type TABLET_DATA_TOMBSTONED (TS 7046cbd82d634976b53fe0cef5788f18 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56268
I20240503 14:16:06.299360 27791 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.300072 30303 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.306696 28451 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.308133 32103 tablet_replica.cc:330] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18: stopping tablet replica
I20240503 14:16:06.308804 32103 raft_consensus.cc:2229] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:06.309373 32103 raft_consensus.cc:2258] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:06.313234 32103 ts_tablet_manager.cc:1875] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:06.319394 28449 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.320427 27659 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.321954 31871 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.322430 31979 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.325006 32103 ts_tablet_manager.cc:1888] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:06.325587 32103 log.cc:1196] T 0e9e6daafc14437faff996cfb40f6dfb P 7046cbd82d634976b53fe0cef5788f18: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/0e9e6daafc14437faff996cfb40f6dfb
I20240503 14:16:06.328183 27507 catalog_manager.cc:4830] TS 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): tablet 0e9e6daafc14437faff996cfb40f6dfb (table rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]) successfully deleted
I20240503 14:16:06.344893 27923 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.345778 31791 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.347608 27923 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.353305 28055 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.354250 30303 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.354960 28186 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.356755 27791 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.356130 30169 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.358702 31871 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.359886 31979 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.363251 31979 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.364095 31979 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.364915 31979 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.370162 32056 ts_tablet_manager.cc:902] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605)
I20240503 14:16:06.383870 31871 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.390151 28451 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.389984 27659 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.392692 32056 tablet_copy_client.cc:308] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.5:35605
I20240503 14:16:06.393895 28208 tablet_copy_service.cc:140] P c91f285b8f4f4a27bcdd0ada303c1422: Received BeginTabletCopySession request for tablet 29945558359248c6a8a4c68952cfcc81 from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:35369)
I20240503 14:16:06.394109 28208 tablet_copy_service.cc:161] P c91f285b8f4f4a27bcdd0ada303c1422: Beginning new tablet copy session on tablet 29945558359248c6a8a4c68952cfcc81 from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:35369: session id = 54dab61f1a204c0da4f56f98b3fd2690-29945558359248c6a8a4c68952cfcc81
I20240503 14:16:06.395391 27515 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:06.397184 28208 tablet_copy_source_session.cc:205] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:06.401172 32056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29945558359248c6a8a4c68952cfcc81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.401378 30168 raft_consensus.cc:1203] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:06.409183 32056 tablet_copy_client.cc:783] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:06.412037 32056 tablet_copy_client.cc:647] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:06.412676 30303 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.413615 31915 raft_consensus.cc:1050] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:16:06.414958 31915 consensus_queue.cc:237] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:06.415279 28186 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.421247 27791 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.428932 28055 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.434810 32058 raft_consensus.cc:1050] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:16:06.436077 32058 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:06.446612 32056 tablet_copy_client.cc:523] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:06.446323 30169 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.452811 32056 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:16:06.457896 30303 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.458410 27659 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.458228 27923 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:06.458943 32049 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.459290 32067 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.460026 27922 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.461757 28317 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.461973 32049 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.462378 27791 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.463076 32052 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.463621 32052 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.464391 28451 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.464980 27661 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.466312 31871 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.473050 32123 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.488078 30304 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.488448 28055 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.490978 30168 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.491279 31979 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.491947 31979 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.492120 28186 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.492291 31935 raft_consensus.cc:1050] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:16:06.493582 31935 consensus_queue.cc:237] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:06.500905 31979 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.501746 31979 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.506016 30169 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.506841 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.509495 32124 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.523263 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e895331729d3400384f20aa3895540de with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:06.529603 28451 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.534634 27661 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.551507 30302 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.551612 28055 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.549750 27518 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.571210 30169 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.587556 27772 tablet_service.cc:1514] Processing DeleteTablet for tablet e895331729d3400384f20aa3895540de with delete_type TABLET_DATA_TOMBSTONED (TS 74318b6270b64efaa79867167fc08bbb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45204
I20240503 14:16:06.591372 28186 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.605125 32141 tablet_replica.cc:330] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb: stopping tablet replica
I20240503 14:16:06.606856 32139 ts_tablet_manager.cc:902] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:16:06.608911 30304 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.609900 32051 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.615960 32058 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.621529 32141 raft_consensus.cc:2229] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:06.624030 32141 raft_consensus.cc:2258] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:06.626530 30304 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.630225 27512 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:06.633986 30168 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.637763 27791 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.639353 32051 raft_consensus.cc:1050] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:16:06.641507 28319 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.640755 32051 consensus_queue.cc:237] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:06.663125 28317 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.663336 30035 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.663906 31210 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.664642 31910 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.670970 27922 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.673343 28186 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.673756 30169 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.673872 27924 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.674654 31910 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.675220 27793 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.676123 31210 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.676734 31210 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.677300 31915 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.678921 27661 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.680518 31915 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.683099 32085 raft_consensus.cc:1050] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:16:06.684509 32085 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:06.692232 31963 tablet_copy_client.cc:308] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:45951
I20240503 14:16:06.693018 28077 tablet_copy_service.cc:140] P 078f8eff84c94dde9705ad1330922a11: Received BeginTabletCopySession request for tablet 9f00380931064f71bc78626af72488f3 from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:38405)
I20240503 14:16:06.693230 28077 tablet_copy_service.cc:161] P 078f8eff84c94dde9705ad1330922a11: Beginning new tablet copy session on tablet 9f00380931064f71bc78626af72488f3 from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:38405: session id = 01173ed91fd341e3a4a4599118167e52-9f00380931064f71bc78626af72488f3
I20240503 14:16:06.696385 28077 tablet_copy_source_session.cc:205] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:06.698961 31963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f00380931064f71bc78626af72488f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.634685 32141 ts_tablet_manager.cc:1875] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:06.709389 31963 tablet_copy_client.cc:783] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:06.713814 31963 tablet_copy_client.cc:647] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:06.717072 31963 tablet_copy_client.cc:523] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:06.724943 31963 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:16:06.725219 30165 raft_consensus.cc:1203] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:06.727880 28451 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.727871 27923 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.728091 28449 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.729835 30036 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.729537 31915 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.731379 28055 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.732434 30036 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.738422 28317 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.742985 32148 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.743745 32146 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 14:16:06.747468 27518 catalog_manager.cc:5484] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.749588 30302 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.750254 27793 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.750731 27659 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.758529 27923 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.761564 28188 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.762523 28055 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.763037 27661 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.764835 30035 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.773793 32145 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.774485 32145 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.777181 27791 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.778649 28055 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.780867 28186 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.781944 32145 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.782052 31935 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.782780 32145 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.783303 32145 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.787771 32145 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.788162 27662 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.788802 30035 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.788884 31958 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.789435 32145 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.789513 31948 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.790704 32145 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.797331 32145 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.799832 32145 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.807536 32141 ts_tablet_manager.cc:1888] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:06.808615 32141 log.cc:1196] T e895331729d3400384f20aa3895540de P 74318b6270b64efaa79867167fc08bbb: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/e895331729d3400384f20aa3895540de
I20240503 14:16:06.812891 27508 catalog_manager.cc:4830] TS 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): tablet e895331729d3400384f20aa3895540de (table rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]) successfully deleted
I20240503 14:16:06.832847 27794 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.833813 31948 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.841714 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.854655 31963 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:06.855233 31963 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:16:06.855597 31963 ts_tablet_manager.cc:1367] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.132s user 0.050s sys 0.009s
I20240503 14:16:06.856546 31963 raft_consensus.cc:348] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:06.856976 31963 raft_consensus.cc:729] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:16:06.857228 31963 consensus_queue.cc:260] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:06.865397 31963 ts_tablet_manager.cc:1398] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240503 14:16:06.867180 28077 tablet_copy_service.cc:342] P 078f8eff84c94dde9705ad1330922a11: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-9f00380931064f71bc78626af72488f3 received from {username='slave'} at 127.10.145.10:38405
I20240503 14:16:06.867355 28077 tablet_copy_service.cc:433] P 078f8eff84c94dde9705ad1330922a11: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-9f00380931064f71bc78626af72488f3 on tablet 9f00380931064f71bc78626af72488f3 with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:16:06.878924 30304 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.880939 31958 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:16:06.884498 28387 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.3:36119, user_credentials={real_user=slave}} blocked reactor thread for 143658us
I20240503 14:16:06.890830 28187 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.891865 31948 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.897225 27791 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.903021 32139 tablet_copy_client.cc:308] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36073
I20240503 14:16:06.910024 28340 tablet_copy_service.cc:140] P acb93936f2f14e628eecb05d7d1f4d91: Received BeginTabletCopySession request for tablet add941da26824cdda7489b5da990e879 from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:33777)
I20240503 14:16:06.910319 28340 tablet_copy_service.cc:161] P acb93936f2f14e628eecb05d7d1f4d91: Beginning new tablet copy session on tablet add941da26824cdda7489b5da990e879 from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:33777: session id = 01173ed91fd341e3a4a4599118167e52-add941da26824cdda7489b5da990e879
I20240503 14:16:06.913825 28340 tablet_copy_source_session.cc:205] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:06.917750 32139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add941da26824cdda7489b5da990e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.919798 27924 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:06.920833 32085 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:06.943506 32148 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:06.904515 28188 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:06.946838 32142 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:06.948575 28056 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.954279 27518 catalog_manager.cc:5484] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:06.959569 28185 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:06.961869 27923 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:06.965862 27518 catalog_manager.cc:5484] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:06.966773 30033 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:06.968631 28186 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:06.969907 30302 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:06.979260 28450 consensus_queue.cc:237] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:06.983714 28319 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:06.983764 30304 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:06.984695 31910 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:06.988289 27926 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:06.992448 31948 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:06.998152 28055 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.000324 30303 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.001956 27662 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.001502 28057 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.002810 32159 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 14:16:07.003928 27993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.9:39057, user_credentials={real_user=slave}} blocked reactor thread for 60090.9us
I20240503 14:16:07.006444 30034 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.010035 27518 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 reported cstate change: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.011272 27793 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.014201 30032 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.016605 27792 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.022728 32026 raft_consensus.cc:1050] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:16:07.023264 28187 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.024559 32026 consensus_queue.cc:237] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:07.024595 27790 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.026235 27661 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.034549 27924 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.040573 28448 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.047742 30169 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.048684 30035 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.049470 32056 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:07.057126 27926 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.058986 30168 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.060254 28185 raft_consensus.cc:1261] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.061372 31948 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.067524 32139 tablet_copy_client.cc:783] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:07.067857 32139 tablet_copy_client.cc:647] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:07.070051 27661 consensus_queue.cc:237] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:07.070709 27662 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.071764 30303 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.072935 27926 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.073626 32183 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.075718 30169 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.078301 31791 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.079023 31791 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.079614 31791 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.081182 28188 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.084466 27793 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.089213 27924 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.099524 32056 tablet_bootstrap.cc:492] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:16:07.098472 27661 consensus_queue.cc:237] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:07.099920 32056 ts_tablet_manager.cc:1367] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.647s user 0.093s sys 0.080s
I20240503 14:16:07.100826 32056 raft_consensus.cc:348] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:07.101151 32056 raft_consensus.cc:729] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:16:07.101490 32056 consensus_queue.cc:260] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:07.102167 32056 ts_tablet_manager.cc:1398] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:07.103041 28208 tablet_copy_service.cc:342] P c91f285b8f4f4a27bcdd0ada303c1422: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-29945558359248c6a8a4c68952cfcc81 received from {username='slave'} at 127.10.145.9:35369
I20240503 14:16:07.103188 28208 tablet_copy_service.cc:433] P c91f285b8f4f4a27bcdd0ada303c1422: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-29945558359248c6a8a4c68952cfcc81 on tablet 29945558359248c6a8a4c68952cfcc81 with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:16:07.108660 32139 tablet_copy_client.cc:523] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:07.111382 30032 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.112039 32148 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.112238 30303 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.112505 27793 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.113340 32139 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: Bootstrap starting.
I20240503 14:16:07.113529 32183 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.114130 28185 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.114182 32183 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.116313 32142 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.116971 32142 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.117417 32159 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.117558 32142 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.118924 31910 raft_consensus.cc:1050] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:16:07.120131 31910 consensus_queue.cc:237] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:07.122817 30304 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.124382 32145 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.125314 32145 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.131415 28185 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.133848 28055 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.134707 32142 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.139613 28057 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.140589 32142 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.143150 27513 catalog_manager.cc:5484] T 77469d6a6610402ca96598bb695224d9 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.145730 27792 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.147941 28449 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.148571 31915 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.148448 28055 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.151206 28448 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.152590 30034 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.155462 28187 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.156112 32148 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.155659 28449 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.157783 31915 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.158526 31915 consensus_queue.cc:1035] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.157433 28451 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.159325 31791 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.160029 32148 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.160921 32146 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.161576 31210 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.166395 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56dad6fc0ba0440aa4660030b1bb9267 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:07.166447 28447 raft_consensus.cc:1261] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.168002 27506 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d9c369355ad47f4843da19dc3ae3cac with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:07.169809 27516 catalog_manager.cc:5484] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:07.170765 32148 consensus_queue.cc:1035] T 77469d6a6610402ca96598bb695224d9 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.170807 27792 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.173542 28319 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.176348 32051 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.180917 30032 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.181502 27661 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.182534 28187 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.189496 27512 catalog_manager.cc:5484] T 0d9c369355ad47f4843da19dc3ae3cac P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.190011 27793 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.192081 27904 tablet_service.cc:1514] Processing DeleteTablet for tablet 56dad6fc0ba0440aa4660030b1bb9267 with delete_type TABLET_DATA_TOMBSTONED (TS a242f99f54b4494893064cfe3030b4ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:46680
I20240503 14:16:07.193742 30304 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.194917 28450 raft_consensus.cc:1261] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.202173 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 77469d6a6610402ca96598bb695224d9 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:07.204025 32202 tablet_replica.cc:330] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad: stopping tablet replica
I20240503 14:16:07.205178 28057 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.206074 27904 tablet_service.cc:1514] Processing DeleteTablet for tablet 77469d6a6610402ca96598bb695224d9 with delete_type TABLET_DATA_TOMBSTONED (TS a242f99f54b4494893064cfe3030b4ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:46680
I20240503 14:16:07.207999 32202 raft_consensus.cc:2229] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:07.217816 30169 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.218701 32202 raft_consensus.cc:2258] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:07.221086 32202 ts_tablet_manager.cc:1875] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:07.221146 28450 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.222247 30303 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.227660 30034 raft_consensus.cc:2932] T 77469d6a6610402ca96598bb695224d9 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.232000 30301 raft_consensus.cc:1203] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:07.235347 27772 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9c369355ad47f4843da19dc3ae3cac with delete_type TABLET_DATA_TOMBSTONED (TS 74318b6270b64efaa79867167fc08bbb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45204
I20240503 14:16:07.236009 32146 consensus_queue.cc:1035] T 56dad6fc0ba0440aa4660030b1bb9267 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.240291 32141 tablet_replica.cc:330] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb: stopping tablet replica
I20240503 14:16:07.240588 32141 raft_consensus.cc:2229] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:07.241978 32141 raft_consensus.cc:2258] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:07.252453 31791 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.255604 32141 ts_tablet_manager.cc:1875] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:07.257515 27922 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.259796 27661 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.260795 31210 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.264650 32141 ts_tablet_manager.cc:1888] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:07.264904 32141 log.cc:1196] T 0d9c369355ad47f4843da19dc3ae3cac P 74318b6270b64efaa79867167fc08bbb: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/0d9c369355ad47f4843da19dc3ae3cac
I20240503 14:16:07.266319 30168 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.266552 27662 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.270339 27518 catalog_manager.cc:5484] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:07.274806 27508 catalog_manager.cc:4830] TS 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): tablet 0d9c369355ad47f4843da19dc3ae3cac (table rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]) successfully deleted
I20240503 14:16:07.279450 28319 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.280560 31210 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.280913 28188 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.286579 27794 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.289183 30304 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.292035 32202 ts_tablet_manager.cc:1888] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:07.304082 28187 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.308820 27792 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.309633 30304 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.312971 28188 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.316777 30032 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.309144 27924 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.321736 32202 log.cc:1196] T 56dad6fc0ba0440aa4660030b1bb9267 P a242f99f54b4494893064cfe3030b4ad: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/56dad6fc0ba0440aa4660030b1bb9267
I20240503 14:16:07.322399 28057 consensus_queue.cc:237] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:07.323050 27508 catalog_manager.cc:4830] TS a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119): tablet 56dad6fc0ba0440aa4660030b1bb9267 (table rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]) successfully deleted
I20240503 14:16:07.323019 27924 raft_consensus.cc:2932] T 0d9c369355ad47f4843da19dc3ae3cac P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.326375 30169 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.332212 27662 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.332618 28450 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.333413 32183 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.343420 28448 raft_consensus.cc:2932] T 56dad6fc0ba0440aa4660030b1bb9267 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:07.350095 31910 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.349442 32202 tablet_replica.cc:330] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad: stopping tablet replica
I20240503 14:16:07.351178 32202 raft_consensus.cc:2229] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:07.351466 32202 raft_consensus.cc:2258] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:07.352669 32201 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.353096 32202 ts_tablet_manager.cc:1875] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:07.353991 32201 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.355577 32201 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.369334 30303 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.369273 31910 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 14:16:07.371109 30169 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.382416 27792 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.383826 31910 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.386543 32139 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:07.386636 32075 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.386994 32139 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: Bootstrap complete.
I20240503 14:16:07.387302 32139 ts_tablet_manager.cc:1367] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: Time spent bootstrapping tablet: real 0.274s user 0.047s sys 0.003s
I20240503 14:16:07.388422 32139 raft_consensus.cc:348] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:07.388743 32139 raft_consensus.cc:729] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01173ed91fd341e3a4a4599118167e52, State: Initialized, Role: LEARNER
I20240503 14:16:07.389045 28188 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.389142 28450 raft_consensus.cc:1261] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.388988 32139 consensus_queue.cc:260] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:07.389650 32139 ts_tablet_manager.cc:1398] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:16:07.390079 30169 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.390972 28340 tablet_copy_service.cc:342] P acb93936f2f14e628eecb05d7d1f4d91: Request end of tablet copy session 01173ed91fd341e3a4a4599118167e52-add941da26824cdda7489b5da990e879 received from {username='slave'} at 127.10.145.10:33777
I20240503 14:16:07.391113 28340 tablet_copy_service.cc:433] P acb93936f2f14e628eecb05d7d1f4d91: ending tablet copy session 01173ed91fd341e3a4a4599118167e52-add941da26824cdda7489b5da990e879 on tablet add941da26824cdda7489b5da990e879 with peer 01173ed91fd341e3a4a4599118167e52
I20240503 14:16:07.392490 32075 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.397361 30304 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.397717 27924 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.398231 28319 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.401700 27662 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.401700 27794 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.404713 32202 ts_tablet_manager.cc:1888] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:07.404852 32202 log.cc:1196] T 77469d6a6610402ca96598bb695224d9 P a242f99f54b4494893064cfe3030b4ad: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/77469d6a6610402ca96598bb695224d9
I20240503 14:16:07.405845 27508 catalog_manager.cc:4830] TS a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119): tablet 77469d6a6610402ca96598bb695224d9 (table rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]) successfully deleted
I20240503 14:16:07.408571 27518 catalog_manager.cc:5484] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 reported cstate change: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.409619 28187 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.415877 32075 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.416819 32075 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.425827 32209 consensus_queue.cc:1035] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.435222 28448 consensus_queue.cc:237] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:07.438477 31791 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.443648 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b178a32f4b24249aeae16cada4a3d56 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:07.445122 28188 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:07.445223 27661 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.446266 28450 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.447082 27518 catalog_manager.cc:5484] T 6b178a32f4b24249aeae16cada4a3d56 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:07.449260 30304 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.450021 32049 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.452629 27904 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b178a32f4b24249aeae16cada4a3d56 with delete_type TABLET_DATA_TOMBSTONED (TS a242f99f54b4494893064cfe3030b4ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:46680
I20240503 14:16:07.453259 32202 tablet_replica.cc:330] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad: stopping tablet replica
I20240503 14:16:07.453568 32202 raft_consensus.cc:2229] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:07.454180 32202 raft_consensus.cc:2258] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:07.455258 28186 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.456287 32202 ts_tablet_manager.cc:1875] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:07.456336 30169 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.461704 30304 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.463639 27794 raft_consensus.cc:2932] T 6b178a32f4b24249aeae16cada4a3d56 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.471356 32202 ts_tablet_manager.cc:1888] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:07.471890 32202 log.cc:1196] T 6b178a32f4b24249aeae16cada4a3d56 P a242f99f54b4494893064cfe3030b4ad: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/6b178a32f4b24249aeae16cada4a3d56
I20240503 14:16:07.473045 27924 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.473551 30165 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.473685 30168 raft_consensus.cc:1203] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:07.474088 27508 catalog_manager.cc:4830] TS a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119): tablet 6b178a32f4b24249aeae16cada4a3d56 (table rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]) successfully deleted
I20240503 14:16:07.474763 27794 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.477877 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.478426 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.478788 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.479298 28319 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:07.483367 28450 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.483428 30032 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.483906 27924 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.484021 32085 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.484720 32085 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.485009 31958 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.485378 30304 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.486081 31958 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.489003 32049 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.491261 31958 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.494642 30168 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.497426 30165 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.497594 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12dae8a392374b91be378ec431838686 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:07.498596 30302 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.498544 31915 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.498132 27792 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.499456 27922 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.501137 28317 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.501175 27661 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.502002 32159 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.502599 32159 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.503798 27924 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.505128 27516 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.508322 27794 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.508940 32159 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.509152 31948 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.509500 32159 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.510413 28186 raft_consensus.cc:1261] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.514797 31915 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.515233 30302 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.516003 27642 tablet_service.cc:1514] Processing DeleteTablet for tablet 12dae8a392374b91be378ec431838686 with delete_type TABLET_DATA_TOMBSTONED (TS 7046cbd82d634976b53fe0cef5788f18 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56268
I20240503 14:16:07.517813 31910 consensus_queue.cc:1035] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.520004 27922 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.520576 27506 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:07.524214 27661 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.525735 28186 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.526587 27794 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.530719 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c23af4dc19c412685927c70cbee7124 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:07.531910 27513 catalog_manager.cc:5484] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 01173ed91fd341e3a4a4599118167e52 reported cstate change: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.531842 28317 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.533869 30302 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.537070 27772 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fb4ed8413f4eceb4348f9d216d3dd7 with delete_type TABLET_DATA_TOMBSTONED (TS 74318b6270b64efaa79867167fc08bbb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45204
I20240503 14:16:07.537492 32141 tablet_replica.cc:330] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb: stopping tablet replica
I20240503 14:16:07.537679 32141 raft_consensus.cc:2229] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:07.538301 32141 raft_consensus.cc:2258] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:07.539356 32141 ts_tablet_manager.cc:1875] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:07.539407 27516 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.539750 30168 raft_consensus.cc:2932] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.542721 30032 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.546115 32141 ts_tablet_manager.cc:1888] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:07.546255 32141 log.cc:1196] T 89fb4ed8413f4eceb4348f9d216d3dd7 P 74318b6270b64efaa79867167fc08bbb: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/89fb4ed8413f4eceb4348f9d216d3dd7
I20240503 14:16:07.546633 28185 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.547257 27508 catalog_manager.cc:4830] TS 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): tablet 89fb4ed8413f4eceb4348f9d216d3dd7 (table rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]) successfully deleted
I20240503 14:16:07.555852 32184 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.557950 27641 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c23af4dc19c412685927c70cbee7124 with delete_type TABLET_DATA_TOMBSTONED (TS 7046cbd82d634976b53fe0cef5788f18 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56268
I20240503 14:16:07.564544 28055 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.565747 32216 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.567879 28055 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.570541 27924 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.577438 28319 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.578166 32216 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.578804 28188 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.580435 28319 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.581576 30302 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.586457 28448 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.595196 32219 tablet_replica.cc:330] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: stopping tablet replica
I20240503 14:16:07.595441 32219 raft_consensus.cc:2229] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:07.597728 32219 raft_consensus.cc:2258] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:07.598752 30304 raft_consensus.cc:1203] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:07.632042 32219 ts_tablet_manager.cc:1875] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:07.655411 32219 ts_tablet_manager.cc:1888] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:07.655684 32219 log.cc:1196] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/12dae8a392374b91be378ec431838686
I20240503 14:16:07.656637 27507 catalog_manager.cc:4830] TS 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): tablet 12dae8a392374b91be378ec431838686 (table rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]) successfully deleted
I20240503 14:16:07.657872 32219 tablet_replica.cc:330] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18: stopping tablet replica
I20240503 14:16:07.658219 32219 raft_consensus.cc:2229] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:07.659338 32219 raft_consensus.cc:2258] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:07.668624 32219 ts_tablet_manager.cc:1875] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:07.675845 32219 ts_tablet_manager.cc:1888] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:07.676095 32219 log.cc:1196] T 4c23af4dc19c412685927c70cbee7124 P 7046cbd82d634976b53fe0cef5788f18: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/4c23af4dc19c412685927c70cbee7124
I20240503 14:16:07.676995 27507 catalog_manager.cc:4830] TS 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): tablet 4c23af4dc19c412685927c70cbee7124 (table rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]) successfully deleted
I20240503 14:16:07.704792 32209 raft_consensus.cc:1050] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:16:07.706219 32209 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:07.727962 28448 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.728107 30304 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.728596 28186 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.728668 31791 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.731333 31791 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.734962 31791 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.735208 27661 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.735239 30168 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.735268 27924 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.735810 32209 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.735901 27792 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.740244 31791 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.740762 32026 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.741313 32075 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.771515 31791 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.775548 27662 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.783206 28448 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.783604 28188 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.788506 30168 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.793500 27924 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.795445 27518 catalog_manager.cc:5484] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.798244 27792 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.811491 30302 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.820891 28055 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:07.826360 28448 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.827155 30168 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.827800 30304 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.828380 27924 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.830251 28188 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.830214 32026 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.830844 32026 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.831357 32026 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.831836 32026 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.832237 32026 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.838127 32183 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.838765 27792 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.838759 27924 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.839015 30168 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.839613 32026 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.841267 27794 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.845034 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9f00380931064f71bc78626af72488f3 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:07.844806 28448 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.845645 27513 catalog_manager.cc:5484] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.847924 30304 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.851359 28188 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:07.853451 27642 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f00380931064f71bc78626af72488f3 with delete_type TABLET_DATA_TOMBSTONED (TS 7046cbd82d634976b53fe0cef5788f18 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56268
I20240503 14:16:07.853884 32219 tablet_replica.cc:330] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18: stopping tablet replica
I20240503 14:16:07.854076 32219 raft_consensus.cc:2229] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:07.854478 32219 raft_consensus.cc:2258] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:07.855337 32219 ts_tablet_manager.cc:1875] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:07.861202 32219 ts_tablet_manager.cc:1888] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:07.861372 32219 log.cc:1196] T 9f00380931064f71bc78626af72488f3 P 7046cbd82d634976b53fe0cef5788f18: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f00380931064f71bc78626af72488f3
I20240503 14:16:07.862185 27507 catalog_manager.cc:4830] TS 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): tablet 9f00380931064f71bc78626af72488f3 (table rebalance_test_table_6 [id=9fb1f7433cf443578aae82759259df41]) successfully deleted
I20240503 14:16:07.896929 32216 raft_consensus.cc:1050] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:16:07.897986 32216 consensus_queue.cc:237] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:07.901365 27924 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.901378 30304 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.902371 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.902467 30168 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.902969 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.903249 28448 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.903388 27794 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.903458 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.903918 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.903965 28055 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.905036 27661 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:07.906301 32217 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.920558 32265 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.921131 32217 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:07.927661 32216 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.932583 28055 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.933517 27662 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.939411 27513 catalog_manager.cc:5484] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.943953 27924 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.944365 27794 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.948033 30168 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.960249 28448 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.970635 28188 consensus_queue.cc:237] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:07.966423 30304 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.974133 27661 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.974836 27924 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.974903 30168 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.975324 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.975875 28055 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.975901 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.976281 28448 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.976569 32265 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.977370 32265 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.977411 30302 raft_consensus.cc:1261] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:07.978904 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.979401 32052 consensus_queue.cc:1035] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:07.982335 32265 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.984789 27924 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.984922 30168 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.986567 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 29945558359248c6a8a4c68952cfcc81 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:07.986735 28055 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.987757 27661 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.990790 27513 catalog_manager.cc:5484] T 29945558359248c6a8a4c68952cfcc81 P 54dab61f1a204c0da4f56f98b3fd2690 reported cstate change: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.991868 28448 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:07.998808 27772 tablet_service.cc:1514] Processing DeleteTablet for tablet 29945558359248c6a8a4c68952cfcc81 with delete_type TABLET_DATA_TOMBSTONED (TS 74318b6270b64efaa79867167fc08bbb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45204
I20240503 14:16:07.999229 32141 tablet_replica.cc:330] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb: stopping tablet replica
I20240503 14:16:07.999429 32141 raft_consensus.cc:2229] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:07.999766 32141 raft_consensus.cc:2258] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:08.001083 32141 ts_tablet_manager.cc:1875] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:08.005462 30302 raft_consensus.cc:2932] T 29945558359248c6a8a4c68952cfcc81 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:08.012003 32141 ts_tablet_manager.cc:1888] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:08.012315 32141 log.cc:1196] T 29945558359248c6a8a4c68952cfcc81 P 74318b6270b64efaa79867167fc08bbb: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/29945558359248c6a8a4c68952cfcc81
I20240503 14:16:08.013962 27508 catalog_manager.cc:4830] TS 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): tablet 29945558359248c6a8a4c68952cfcc81 (table rebalance_test_table_3 [id=1a899dedd0694cc99c4b77cb569875f1]) successfully deleted
I20240503 14:16:08.053205 31958 raft_consensus.cc:1050] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: attempting to promote NON_VOTER 01173ed91fd341e3a4a4599118167e52 to VOTER
I20240503 14:16:08.054210 31958 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:08.058871 30168 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:08.059128 30302 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 LEARNER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:08.059530 27792 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:08.059739 27924 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:08.060438 32218 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:08.060760 32184 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:08.062258 27661 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:08.062518 32218 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:08.063041 32218 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:08.063365 32184 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:08.096091 32218 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.097590 27662 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.100778 28448 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:08.103924 27516 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:08.104126 27924 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.106303 28188 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:08.107035 27794 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.107272 30168 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.108482 30304 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.114951 32184 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:08.115679 32184 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:08.138006 28448 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.139261 28188 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.724045 28319 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } }
I20240503 14:16:08.726714 30302 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:08.727082 27922 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:08.727658 30169 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:08.728044 27794 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:08.728190 32218 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:08.728354 28448 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:08.728374 28188 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:08.728787 32218 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:08.729132 32305 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:08.729617 32333 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:08.729938 32332 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:08.730211 32305 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:08.733749 32218 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.734469 30169 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.737706 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet add941da26824cdda7489b5da990e879 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:08.737104 28448 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.739629 27922 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.739923 30302 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.739920 27794 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.745682 27513 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:08.750064 28188 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } }
I20240503 14:16:08.759384 27642 tablet_service.cc:1514] Processing DeleteTablet for tablet add941da26824cdda7489b5da990e879 with delete_type TABLET_DATA_TOMBSTONED (TS 7046cbd82d634976b53fe0cef5788f18 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56268
I20240503 14:16:08.760094 32340 tablet_replica.cc:330] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18: stopping tablet replica
I20240503 14:16:08.760305 32340 raft_consensus.cc:2229] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:08.760645 32340 raft_consensus.cc:2258] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:08.762779 32340 ts_tablet_manager.cc:1875] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:08.767642 32340 ts_tablet_manager.cc:1888] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:08.767889 32340 log.cc:1196] T add941da26824cdda7489b5da990e879 P 7046cbd82d634976b53fe0cef5788f18: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-0/wal/wals/add941da26824cdda7489b5da990e879
I20240503 14:16:08.768860 27507 catalog_manager.cc:4830] TS 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): tablet add941da26824cdda7489b5da990e879 (table rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]) successfully deleted
I20240503 14:16:08.905965 30284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:08.919795 28035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:08.920147 30149 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:08.922847 27642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:08.953429 28430 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:08.961503 27772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:08.994597 27904 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:09.020314 28168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:09.039060 28299 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:09.067723 30016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.163465 27792 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:09.167331 28188 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.167457 28319 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.168602 28448 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:16:09.168676 27730 consensus_peers.cc:485] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d84acfe4f36f4193970fd2230229a41f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:09.169107 32241 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:09.169515 32397 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:09.170284 32398 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240503 14:16:09.171798 27661 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.172237 30302 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.173945 27922 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.174934 32241 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:09.175554 32241 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:09.177873 32241 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20240503 14:16:09.187125 32400 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:09.187397 28319 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:09.187860 27922 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:09.194765 30302 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:09.195012 28188 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:09.196614 27516 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:09.202776 27661 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:09.209409 28448 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:09.224045 28319 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:09.227849 28448 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.228292 28188 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.228529 27661 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.229159 27792 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.229521 30032 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.229846 32333 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:09.230279 32334 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:09.230976 32332 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:09.231686 27922 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:09.231741 32332 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:09.232260 28256 consensus_peers.cc:485] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 233ee592f6e843c8a68a5170e0f9d66d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:09.232257 32332 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240503 14:16:09.233215 32218 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:09.243007 32334 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:09.244131 28188 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:09.244131 27794 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:09.250198 27516 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:09.253639 28448 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:09.255316 30032 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:09.252928 27661 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:09.262702 27922 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:09.602841 32444 ts_tablet_manager.cc:902] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859)
I20240503 14:16:09.610662 32444 tablet_copy_client.cc:308] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:38859
I20240503 14:16:09.611625 27814 tablet_copy_service.cc:140] P 74318b6270b64efaa79867167fc08bbb: Received BeginTabletCopySession request for tablet d84acfe4f36f4193970fd2230229a41f from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:51227)
I20240503 14:16:09.611812 27814 tablet_copy_service.cc:161] P 74318b6270b64efaa79867167fc08bbb: Beginning new tablet copy session on tablet d84acfe4f36f4193970fd2230229a41f from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:51227: session id = e938d875af584f1f88a2d4b50c589268-d84acfe4f36f4193970fd2230229a41f
I20240503 14:16:09.614971 27814 tablet_copy_source_session.cc:205] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:09.616560 32444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84acfe4f36f4193970fd2230229a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:09.624089 32444 tablet_copy_client.cc:783] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:09.625351 32444 tablet_copy_client.cc:647] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:09.627489 32444 tablet_copy_client.cc:523] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:09.631150 32444 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:16:09.691797 32450 ts_tablet_manager.cc:902] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:16:09.694424 32450 tablet_copy_client.cc:308] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36073
I20240503 14:16:09.695303 28340 tablet_copy_service.cc:140] P acb93936f2f14e628eecb05d7d1f4d91: Received BeginTabletCopySession request for tablet 233ee592f6e843c8a68a5170e0f9d66d from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:41081)
I20240503 14:16:09.695447 28340 tablet_copy_service.cc:161] P acb93936f2f14e628eecb05d7d1f4d91: Beginning new tablet copy session on tablet 233ee592f6e843c8a68a5170e0f9d66d from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:41081: session id = 54dab61f1a204c0da4f56f98b3fd2690-233ee592f6e843c8a68a5170e0f9d66d
I20240503 14:16:09.697685 28340 tablet_copy_source_session.cc:205] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:09.699189 32450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ee592f6e843c8a68a5170e0f9d66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:09.704613 32450 tablet_copy_client.cc:783] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:09.704916 32450 tablet_copy_client.cc:647] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:09.705252 32444 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:09.705729 32444 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:16:09.706039 32444 ts_tablet_manager.cc:1367] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.075s user 0.035s sys 0.010s
I20240503 14:16:09.706823 32444 raft_consensus.cc:348] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:09.707041 32450 tablet_copy_client.cc:523] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:09.707134 32444 raft_consensus.cc:729] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:16:09.707379 32444 consensus_queue.cc:260] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:09.708057 32444 ts_tablet_manager.cc:1398] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:16:09.708982 27814 tablet_copy_service.cc:342] P 74318b6270b64efaa79867167fc08bbb: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-d84acfe4f36f4193970fd2230229a41f received from {username='slave'} at 127.10.145.8:51227
I20240503 14:16:09.709120 27814 tablet_copy_service.cc:433] P 74318b6270b64efaa79867167fc08bbb: ending tablet copy session e938d875af584f1f88a2d4b50c589268-d84acfe4f36f4193970fd2230229a41f on tablet d84acfe4f36f4193970fd2230229a41f with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:16:09.714958 32450 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:16:09.769174 32450 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:09.769639 32450 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:16:09.769945 32450 ts_tablet_manager.cc:1367] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.055s user 0.041s sys 0.009s
I20240503 14:16:09.770772 32450 raft_consensus.cc:348] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:09.771071 32450 raft_consensus.cc:729] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:16:09.771311 32450 consensus_queue.cc:260] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:09.772220 32450 ts_tablet_manager.cc:1398] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:16:09.773116 28340 tablet_copy_service.cc:342] P acb93936f2f14e628eecb05d7d1f4d91: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-233ee592f6e843c8a68a5170e0f9d66d received from {username='slave'} at 127.10.145.9:41081
I20240503 14:16:09.773262 28340 tablet_copy_service.cc:433] P acb93936f2f14e628eecb05d7d1f4d91: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-233ee592f6e843c8a68a5170e0f9d66d on tablet 233ee592f6e843c8a68a5170e0f9d66d with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:16:10.113924 30032 raft_consensus.cc:1203] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:10.129949 30168 raft_consensus.cc:1203] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:10.626648 32334 raft_consensus.cc:1050] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:16:10.627825 32334 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:10.639040 30032 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.639215 28448 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.641149 27661 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.641507 32334 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.642138 32480 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.642344 30168 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.642935 28188 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.643581 27922 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.644733 27792 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.645017 32480 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.645387 32505 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.650256 32334 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.650589 32480 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.651204 32334 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.668458 32512 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:10.671707 27661 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:10.673802 27922 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:10.674166 28186 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:10.677805 30168 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:10.677846 30032 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:10.678045 27794 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:10.683501 27516 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:10.697897 28448 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:10.710741 32479 raft_consensus.cc:1050] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:16:10.711999 32479 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:10.725697 30032 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.726586 32398 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.730124 28448 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.731300 28319 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.732158 32398 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.732879 32398 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.741753 28188 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.743243 32398 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.744875 27661 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.749984 32532 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.752707 30302 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.753873 27922 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:10.756191 32537 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 14:16:10.756989 32537 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:10.757866 32537 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.762562 28448 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.762588 28319 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.762533 30032 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.763592 27515 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:10.773468 27661 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.774941 28188 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.776938 27922 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.784720 30302 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.788965 27794 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:10.791869 30302 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:10.792124 28448 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:10.792416 32538 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:10.792752 28319 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:10.793423 32398 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:10.793783 27661 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:10.793994 32398 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:10.794863 30032 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:10.795094 28188 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:10.795421 32398 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:10.796123 32538 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:10.796564 32398 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:10.802078 32537 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.804136 30034 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.804375 30302 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.805930 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d84acfe4f36f4193970fd2230229a41f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:10.806437 28319 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.806949 27662 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.807505 28448 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.812148 28188 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:10.812600 27515 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 9 to 10, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:10.819097 27904 tablet_service.cc:1514] Processing DeleteTablet for tablet d84acfe4f36f4193970fd2230229a41f with delete_type TABLET_DATA_TOMBSTONED (TS a242f99f54b4494893064cfe3030b4ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:46680
I20240503 14:16:10.824443 32544 tablet_replica.cc:330] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad: stopping tablet replica
I20240503 14:16:10.824707 32544 raft_consensus.cc:2229] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:10.825732 32544 raft_consensus.cc:2258] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:10.826994 32544 ts_tablet_manager.cc:1875] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:10.832064 32544 ts_tablet_manager.cc:1888] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:10.832227 32544 log.cc:1196] T d84acfe4f36f4193970fd2230229a41f P a242f99f54b4494893064cfe3030b4ad: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/d84acfe4f36f4193970fd2230229a41f
I20240503 14:16:10.833119 27508 catalog_manager.cc:4830] TS a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119): tablet d84acfe4f36f4193970fd2230229a41f (table rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]) successfully deleted
I20240503 14:16:11.206650 28319 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:11.209812 30169 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:11.211009 27661 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:11.212767 32561 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240503 14:16:11.213548 32505 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:11.213820 28186 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:11.215008 27922 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:11.215483 28448 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:11.215608 32505 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:11.216151 32505 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:11.216943 30032 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:11.219656 32505 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:11.220181 32565 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:11.220934 32565 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:11.222474 30168 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:11.224586 28186 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:11.227092 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 233ee592f6e843c8a68a5170e0f9d66d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:11.228250 28448 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:11.229498 27518 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:11.230158 27661 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:11.228250 27922 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:11.236570 27772 tablet_service.cc:1514] Processing DeleteTablet for tablet 233ee592f6e843c8a68a5170e0f9d66d with delete_type TABLET_DATA_TOMBSTONED (TS 74318b6270b64efaa79867167fc08bbb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45204
I20240503 14:16:11.239722 30032 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 74318b6270b64efaa79867167fc08bbb (127.10.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:11.251241 32567 tablet_replica.cc:330] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb: stopping tablet replica
I20240503 14:16:11.251555 32567 raft_consensus.cc:2229] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:11.252214 32567 raft_consensus.cc:2258] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:11.253326 32567 ts_tablet_manager.cc:1875] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:11.259923 32567 ts_tablet_manager.cc:1888] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:11.260102 32567 log.cc:1196] T 233ee592f6e843c8a68a5170e0f9d66d P 74318b6270b64efaa79867167fc08bbb: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-1/wal/wals/233ee592f6e843c8a68a5170e0f9d66d
I20240503 14:16:11.261016 27508 catalog_manager.cc:4830] TS 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859): tablet 233ee592f6e843c8a68a5170e0f9d66d (table rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]) successfully deleted
I20240503 14:16:11.397713 30284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.409788 28035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:11.418665 30149 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.462332 27642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:11.486800 28299 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:11.500813 27772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:11.561327 28430 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:11.570510 28168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:11.571777 27904 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:11.632100 30016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.935570 30284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.936971 27642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:11.965689 30149 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.967942 28430 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:11.968223 27772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:11.969305 27904 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:12.020092 28299 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:12.054569 28035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:12.068810 28168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:12.069883 30016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:16:12.089620 27596 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.7:43675, user_credentials={real_user=slave}} blocked reactor thread for 56797.1us
I20240503 14:16:12.232146 28319 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:12.235917 27794 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:12.242173 30032 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 14:16:12.244128 27730 consensus_peers.cc:485] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb -> Peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): Couldn't send request to peer 078f8eff84c94dde9705ad1330922a11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:12.244621 32538 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.255553 30302 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.256224 28449 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.256539 28318 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.256731 32694 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.257705 32694 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.258308 32694 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.264498 28186 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } }
I20240503 14:16:12.280792 32538 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:12.286561 28449 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
W20240503 14:16:12.287307 28123 consensus_peers.cc:485] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 -> Peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261): Couldn't send request to peer 7046cbd82d634976b53fe0cef5788f18. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:12.291826 30304 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.292804 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.292728 27520 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:12.296114 30303 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:12.302255 27922 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.302409 30168 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.302801 27792 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.303784 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.304443 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.310830 30032 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.311106 27922 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.312018 32562 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.323436 28449 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.324568 30302 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 14:16:12.328490 28256 consensus_peers.cc:485] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 -> Peer 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): Couldn't send request to peer 54dab61f1a204c0da4f56f98b3fd2690. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c23af4dc19c412685927c70cbee7124. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:12.329030 32710 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.329535 27792 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.329917 32710 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.331004 32711 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.336248 32710 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.342993 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.347698 28055 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.357547 28186 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.358551 32710 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.360769 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.362798 28319 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.372591 32058 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.377451 32058 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } } }
I20240503 14:16:12.398509 28318 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } } }
I20240503 14:16:12.403528 32710 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:12.405906 27520 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 10 to 11, NON_VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) added. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:12.414180 27793 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:12.421627 30302 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } } }
I20240503 14:16:12.421648 27924 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } } }
I20240503 14:16:12.422987 27518 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 10 to 11, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:12.427521 30168 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } } }
I20240503 14:16:12.429656 30304 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:12.434512 30032 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:12.437637 28449 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:12.436811 27922 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:12.457073 28055 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } } }
I20240503 14:16:12.458669 28187 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.458295 27793 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } } }
I20240503 14:16:12.459343 32694 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.460996 27661 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.463181 32694 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.476760 28186 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } } }
I20240503 14:16:12.486240 28318 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:12.490478 30034 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:12.538398 27922 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
W20240503 14:16:12.551225 27861 consensus_peers.cc:485] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad -> Peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Couldn't send request to peer acb93936f2f14e628eecb05d7d1f4d91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:12.553263 30304 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.553457 30168 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.554478 28055 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.554414 32221 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.554687 28186 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.555258 27661 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:12.559342 32221 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.559765 28449 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.560042 32221 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.561216 32745 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.561102 28187 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:12.562777 32221 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.591558 27661 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.596608 32745 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 14:16:12.606906 27862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.10:40989, user_credentials={real_user=slave}} blocked reactor thread for 45682.6us
I20240503 14:16:12.627202 28318 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:12.644018 32745 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.650187 28055 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.651041 30304 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.651110 30168 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.651939 32711 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.652520 27793 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.652581 32711 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.653853 27922 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 14:16:12.654668 28256 consensus_peers.cc:485] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: add941da26824cdda7489b5da990e879. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:12.655002 32712 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.657068 28449 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.658823 30302 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.661494 32712 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.663828 28186 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.667744 32711 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.668397 28187 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:12.671505 32712 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:12.674551 27661 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.680195 27518 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 reported cstate change: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.690120 30168 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.716048 32712 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.719848 30304 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.722579 27922 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.725064 30168 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.723062 28449 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.726007 27793 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.731231 27521 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 10 to 11, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:12.735667 28187 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.738713 28448 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.744910 300 ts_tablet_manager.cc:902] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Initiating tablet copy from peer 74318b6270b64efaa79867167fc08bbb (127.10.145.2:38859)
I20240503 14:16:12.758272 300 tablet_copy_client.cc:308] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.2:38859
I20240503 14:16:12.759495 27814 tablet_copy_service.cc:140] P 74318b6270b64efaa79867167fc08bbb: Received BeginTabletCopySession request for tablet d84acfe4f36f4193970fd2230229a41f from peer 078f8eff84c94dde9705ad1330922a11 ({username='slave'} at 127.10.145.4:36457)
I20240503 14:16:12.759728 27814 tablet_copy_service.cc:161] P 74318b6270b64efaa79867167fc08bbb: Beginning new tablet copy session on tablet d84acfe4f36f4193970fd2230229a41f from peer 078f8eff84c94dde9705ad1330922a11 at {username='slave'} at 127.10.145.4:36457: session id = 078f8eff84c94dde9705ad1330922a11-d84acfe4f36f4193970fd2230229a41f
I20240503 14:16:12.763002 27814 tablet_copy_source_session.cc:205] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:12.766387 300 ts_tablet_manager.cc:1875] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:12.776917 300 ts_tablet_manager.cc:1888] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 14:16:12.777261 300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84acfe4f36f4193970fd2230229a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:12.781038 300 tablet_copy_client.cc:783] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:12.781388 300 tablet_copy_client.cc:647] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:12.802531 300 tablet_copy_client.cc:523] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:12.803320 28449 consensus_queue.cc:237] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
I20240503 14:16:12.806139 300 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:16:12.814589 30032 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:16:12.814833 28386 consensus_peers.cc:485] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 -> Peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Couldn't send request to peer acb93936f2f14e628eecb05d7d1f4d91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:12.815591 28055 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.815719 32478 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.816470 28187 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.817678 27661 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.818655 32478 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.821893 27922 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.822141 27793 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.823354 32701 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.824018 32701 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.824416 32478 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.830586 27659 consensus_queue.cc:237] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
W20240503 14:16:12.834970 27598 consensus_peers.cc:485] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f15a6ff6acc242258172e585b1ea34c6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:12.836056 28318 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.836258 28187 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.836961 32491 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.837407 27922 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.837677 32491 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.838030 32675 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.844882 32701 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.846853 27793 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.846830 30168 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.848297 330 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.848847 32675 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.849499 32675 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.853893 28449 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:12.865099 32701 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.875972 28186 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.878338 30032 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.880338 338 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:12.883103 27924 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.883531 27521 catalog_manager.cc:5484] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.886790 27520 catalog_manager.cc:5484] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 reported cstate change: config changed from index 7 to 8, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.891697 300 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:12.892318 300 tablet_bootstrap.cc:492] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Bootstrap complete.
I20240503 14:16:12.892647 300 ts_tablet_manager.cc:1367] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.087s user 0.051s sys 0.007s
I20240503 14:16:12.894846 300 raft_consensus.cc:348] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:12.895520 300 raft_consensus.cc:729] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: LEARNER
I20240503 14:16:12.895757 300 consensus_queue.cc:260] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:12.896620 300 ts_tablet_manager.cc:1398] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:16:12.897504 27814 tablet_copy_service.cc:342] P 74318b6270b64efaa79867167fc08bbb: Request end of tablet copy session 078f8eff84c94dde9705ad1330922a11-d84acfe4f36f4193970fd2230229a41f received from {username='slave'} at 127.10.145.4:36457
I20240503 14:16:12.897634 27814 tablet_copy_service.cc:433] P 74318b6270b64efaa79867167fc08bbb: ending tablet copy session 078f8eff84c94dde9705ad1330922a11-d84acfe4f36f4193970fd2230229a41f on tablet d84acfe4f36f4193970fd2230229a41f with peer 078f8eff84c94dde9705ad1330922a11
I20240503 14:16:12.898687 27661 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.902048 28055 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.916779 343 ts_tablet_manager.cc:902] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690: Initiating tablet copy from peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:16:12.916667 27926 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.923597 28187 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.929539 27794 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.936638 27793 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:12.943454 28318 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.945734 30168 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.949116 324 ts_tablet_manager.cc:902] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Initiating tablet copy from peer c91f285b8f4f4a27bcdd0ada303c1422 (127.10.145.5:35605)
I20240503 14:16:12.949851 343 tablet_copy_client.cc:308] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36073
I20240503 14:16:12.950688 28340 tablet_copy_service.cc:140] P acb93936f2f14e628eecb05d7d1f4d91: Received BeginTabletCopySession request for tablet 4c23af4dc19c412685927c70cbee7124 from peer 54dab61f1a204c0da4f56f98b3fd2690 ({username='slave'} at 127.10.145.9:41081)
I20240503 14:16:12.950870 28340 tablet_copy_service.cc:161] P acb93936f2f14e628eecb05d7d1f4d91: Beginning new tablet copy session on tablet 4c23af4dc19c412685927c70cbee7124 from peer 54dab61f1a204c0da4f56f98b3fd2690 at {username='slave'} at 127.10.145.9:41081: session id = 54dab61f1a204c0da4f56f98b3fd2690-4c23af4dc19c412685927c70cbee7124
I20240503 14:16:12.953687 28340 tablet_copy_source_session.cc:205] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:12.954862 28449 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:12.957340 343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23af4dc19c412685927c70cbee7124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:12.964529 343 tablet_copy_client.cc:783] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:12.965579 343 tablet_copy_client.cc:647] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:12.967589 343 tablet_copy_client.cc:523] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:12.970649 343 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap starting.
I20240503 14:16:12.990562 28318 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:13.004298 27662 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.006099 32712 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.006353 324 tablet_copy_client.cc:308] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.5:35605
I20240503 14:16:13.009755 28208 tablet_copy_service.cc:140] P c91f285b8f4f4a27bcdd0ada303c1422: Received BeginTabletCopySession request for tablet 12dae8a392374b91be378ec431838686 from peer 7046cbd82d634976b53fe0cef5788f18 ({username='slave'} at 127.10.145.1:44309)
I20240503 14:16:13.009963 28208 tablet_copy_service.cc:161] P c91f285b8f4f4a27bcdd0ada303c1422: Beginning new tablet copy session on tablet 12dae8a392374b91be378ec431838686 from peer 7046cbd82d634976b53fe0cef5788f18 at {username='slave'} at 127.10.145.1:44309: session id = 7046cbd82d634976b53fe0cef5788f18-12dae8a392374b91be378ec431838686
I20240503 14:16:13.015581 28208 tablet_copy_source_session.cc:205] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:13.015650 28449 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.016336 32712 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.020725 324 ts_tablet_manager.cc:1875] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:13.024641 30168 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.025930 30032 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.026544 27922 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.027215 32711 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.028793 28186 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 14:16:13.035393 28256 consensus_peers.cc:485] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 -> Peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): Couldn't send request to peer 078f8eff84c94dde9705ad1330922a11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:13.035799 32711 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.037870 32712 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.038156 324 ts_tablet_manager.cc:1888] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 14:16:13.038566 324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dae8a392374b91be378ec431838686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:13.039422 32712 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.042351 324 tablet_copy_client.cc:783] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:13.042762 324 tablet_copy_client.cc:647] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:13.047055 324 tablet_copy_client.cc:523] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:13.050946 324 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap starting.
I20240503 14:16:13.055228 371 ts_tablet_manager.cc:902] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Initiating tablet copy from peer a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:16:13.062395 343 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:13.062888 343 tablet_bootstrap.cc:492] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690: Bootstrap complete.
I20240503 14:16:13.063207 343 ts_tablet_manager.cc:1367] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent bootstrapping tablet: real 0.093s user 0.052s sys 0.019s
I20240503 14:16:13.064036 343 raft_consensus.cc:348] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:13.064388 343 raft_consensus.cc:729] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dab61f1a204c0da4f56f98b3fd2690, State: Initialized, Role: LEARNER
I20240503 14:16:13.064613 343 consensus_queue.cc:260] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: NON_VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: true } }
I20240503 14:16:13.070475 343 ts_tablet_manager.cc:1398] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:16:13.072283 28340 tablet_copy_service.cc:342] P acb93936f2f14e628eecb05d7d1f4d91: Request end of tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-4c23af4dc19c412685927c70cbee7124 received from {username='slave'} at 127.10.145.9:41081
I20240503 14:16:13.072438 28340 tablet_copy_service.cc:433] P acb93936f2f14e628eecb05d7d1f4d91: ending tablet copy session 54dab61f1a204c0da4f56f98b3fd2690-4c23af4dc19c412685927c70cbee7124 on tablet 4c23af4dc19c412685927c70cbee7124 with peer 54dab61f1a204c0da4f56f98b3fd2690
I20240503 14:16:13.074972 32711 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.079514 371 tablet_copy_client.cc:308] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:36119
I20240503 14:16:13.081532 28449 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.086908 27662 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.092885 27520 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 reported cstate change: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:13.100358 27946 tablet_copy_service.cc:140] P a242f99f54b4494893064cfe3030b4ad: Received BeginTabletCopySession request for tablet e895331729d3400384f20aa3895540de from peer acb93936f2f14e628eecb05d7d1f4d91 ({username='slave'} at 127.10.145.6:42479)
I20240503 14:16:13.100612 27946 tablet_copy_service.cc:161] P a242f99f54b4494893064cfe3030b4ad: Beginning new tablet copy session on tablet e895331729d3400384f20aa3895540de from peer acb93936f2f14e628eecb05d7d1f4d91 at {username='slave'} at 127.10.145.6:42479: session id = acb93936f2f14e628eecb05d7d1f4d91-e895331729d3400384f20aa3895540de
I20240503 14:16:13.104233 27946 tablet_copy_source_session.cc:205] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:13.107743 30168 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.116940 371 ts_tablet_manager.cc:1875] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:13.123620 371 ts_tablet_manager.cc:1888] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 14:16:13.124001 371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e895331729d3400384f20aa3895540de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:13.127770 371 tablet_copy_client.cc:783] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:13.128119 371 tablet_copy_client.cc:647] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:13.139322 371 tablet_copy_client.cc:523] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:13.146778 371 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:16:13.155578 28186 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.165793 27924 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.174685 30034 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.193073 324 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:13.193743 324 tablet_bootstrap.cc:492] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Bootstrap complete.
W20240503 14:16:13.197876 27993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.9:39057, user_credentials={real_user=slave}} blocked reactor thread for 159601us
I20240503 14:16:13.200433 324 ts_tablet_manager.cc:1367] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Time spent bootstrapping tablet: real 0.150s user 0.048s sys 0.026s
I20240503 14:16:13.201581 324 raft_consensus.cc:348] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } }
I20240503 14:16:13.202052 324 raft_consensus.cc:729] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7046cbd82d634976b53fe0cef5788f18, State: Initialized, Role: LEARNER
I20240503 14:16:13.202380 324 consensus_queue.cc:260] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: NON_VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: true } }
I20240503 14:16:13.207080 324 ts_tablet_manager.cc:1398] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:16:13.208096 28208 tablet_copy_service.cc:342] P c91f285b8f4f4a27bcdd0ada303c1422: Request end of tablet copy session 7046cbd82d634976b53fe0cef5788f18-12dae8a392374b91be378ec431838686 received from {username='slave'} at 127.10.145.1:44309
I20240503 14:16:13.208288 28208 tablet_copy_service.cc:433] P c91f285b8f4f4a27bcdd0ada303c1422: ending tablet copy session 7046cbd82d634976b53fe0cef5788f18-12dae8a392374b91be378ec431838686 on tablet 12dae8a392374b91be378ec431838686 with peer 7046cbd82d634976b53fe0cef5788f18
I20240503 14:16:13.214282 28057 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
W20240503 14:16:13.229099 27992 consensus_peers.cc:485] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 -> Peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Couldn't send request to peer acb93936f2f14e628eecb05d7d1f4d91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:13.230309 28449 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.231221 28187 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.231259 30304 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.231907 377 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.233748 377 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.233937 30168 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.234284 27922 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.234696 377 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.235311 27662 raft_consensus.cc:1203] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 14:16:13.236613 27794 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.243268 377 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.244812 372 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.246184 372 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.265604 371 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:13.266307 371 tablet_bootstrap.cc:492] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap complete.
I20240503 14:16:13.266728 371 ts_tablet_manager.cc:1367] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.120s user 0.062s sys 0.036s
I20240503 14:16:13.267711 371 raft_consensus.cc:348] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
I20240503 14:16:13.268198 371 raft_consensus.cc:729] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: LEARNER
I20240503 14:16:13.268543 371 consensus_queue.cc:260] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
I20240503 14:16:13.270900 371 ts_tablet_manager.cc:1398] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:16:13.271775 27946 tablet_copy_service.cc:342] P a242f99f54b4494893064cfe3030b4ad: Request end of tablet copy session acb93936f2f14e628eecb05d7d1f4d91-e895331729d3400384f20aa3895540de received from {username='slave'} at 127.10.145.6:42479
I20240503 14:16:13.271914 27946 tablet_copy_service.cc:433] P a242f99f54b4494893064cfe3030b4ad: ending tablet copy session acb93936f2f14e628eecb05d7d1f4d91-e895331729d3400384f20aa3895540de on tablet e895331729d3400384f20aa3895540de with peer acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:16:13.276255 371 ts_tablet_manager.cc:902] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Initiating tablet copy from peer 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675)
W20240503 14:16:13.281108 396 log.cc:924] Time spent T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240503 14:16:13.287119 405 ts_tablet_manager.cc:902] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:16:13.289624 372 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.294200 28449 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.297183 27513 catalog_manager.cc:5484] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:13.312374 28057 raft_consensus.cc:1203] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 14:16:13.319645 27922 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.321058 408 ts_tablet_manager.cc:902] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268: Initiating tablet copy from peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:16:13.326157 27793 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.332604 30169 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.334226 30302 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.348773 28187 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.372372 405 tablet_copy_client.cc:308] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36073
I20240503 14:16:13.373802 28340 tablet_copy_service.cc:140] P acb93936f2f14e628eecb05d7d1f4d91: Received BeginTabletCopySession request for tablet add941da26824cdda7489b5da990e879 from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:54229)
I20240503 14:16:13.374027 28340 tablet_copy_service.cc:161] P acb93936f2f14e628eecb05d7d1f4d91: Beginning new tablet copy session on tablet add941da26824cdda7489b5da990e879 from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:54229: session id = e938d875af584f1f88a2d4b50c589268-add941da26824cdda7489b5da990e879
I20240503 14:16:13.376938 28340 tablet_copy_source_session.cc:205] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:13.378768 405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add941da26824cdda7489b5da990e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:13.387245 371 tablet_copy_client.cc:308] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.7:43675
I20240503 14:16:13.388200 28471 tablet_copy_service.cc:140] P 9231e715a4e342a197d6c5456ba6b399: Received BeginTabletCopySession request for tablet 48a05a802960425383984c379e238682 from peer acb93936f2f14e628eecb05d7d1f4d91 ({username='slave'} at 127.10.145.6:40833)
I20240503 14:16:13.388386 28471 tablet_copy_service.cc:161] P 9231e715a4e342a197d6c5456ba6b399: Beginning new tablet copy session on tablet 48a05a802960425383984c379e238682 from peer acb93936f2f14e628eecb05d7d1f4d91 at {username='slave'} at 127.10.145.6:40833: session id = acb93936f2f14e628eecb05d7d1f4d91-48a05a802960425383984c379e238682
I20240503 14:16:13.391182 28471 tablet_copy_source_session.cc:205] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:13.392405 300 ts_tablet_manager.cc:902] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Initiating tablet copy from peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073)
I20240503 14:16:13.398905 30168 raft_consensus.cc:1203] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 14:16:13.398994 405 tablet_copy_client.cc:783] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:13.399260 405 tablet_copy_client.cc:647] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:13.401716 408 tablet_copy_client.cc:308] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:35261
I20240503 14:16:13.402730 27682 tablet_copy_service.cc:140] P 7046cbd82d634976b53fe0cef5788f18: Received BeginTabletCopySession request for tablet f15a6ff6acc242258172e585b1ea34c6 from peer e938d875af584f1f88a2d4b50c589268 ({username='slave'} at 127.10.145.8:47681)
I20240503 14:16:13.402933 27682 tablet_copy_service.cc:161] P 7046cbd82d634976b53fe0cef5788f18: Beginning new tablet copy session on tablet f15a6ff6acc242258172e585b1ea34c6 from peer e938d875af584f1f88a2d4b50c589268 at {username='slave'} at 127.10.145.8:47681: session id = e938d875af584f1f88a2d4b50c589268-f15a6ff6acc242258172e585b1ea34c6
I20240503 14:16:13.403905 27922 consensus_queue.cc:237] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:13.405992 27682 tablet_copy_source_session.cc:205] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:13.409055 405 tablet_copy_client.cc:523] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 14:16:13.410694 27862 consensus_peers.cc:485] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad -> Peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): Couldn't send request to peer 078f8eff84c94dde9705ad1330922a11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:13.411664 30034 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:13.412429 405 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:16:13.412427 371 ts_tablet_manager.cc:1875] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:13.412743 32764 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:13.414177 27662 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:13.416056 408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15a6ff6acc242258172e585b1ea34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:13.416285 32764 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:13.424800 27794 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:13.424867 408 tablet_copy_client.cc:783] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:13.425339 408 tablet_copy_client.cc:647] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:13.425590 32764 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:13.427523 408 tablet_copy_client.cc:523] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:13.432310 408 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268: Bootstrap starting.
I20240503 14:16:13.433712 28448 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:13.434828 32764 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:13.439021 28318 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:13.439163 28187 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:13.439738 435 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:13.444566 32764 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:13.452536 371 ts_tablet_manager.cc:1888] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 14:16:13.452862 371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a05a802960425383984c379e238682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:13.458969 435 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.465950 27794 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.476764 371 tablet_copy_client.cc:783] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:13.477030 371 tablet_copy_client.cc:647] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:13.485309 30034 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.483811 28448 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.485865 27513 catalog_manager.cc:5484] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:13.507002 300 tablet_copy_client.cc:308] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.6:36073
I20240503 14:16:13.507773 28340 tablet_copy_service.cc:140] P acb93936f2f14e628eecb05d7d1f4d91: Received BeginTabletCopySession request for tablet 233ee592f6e843c8a68a5170e0f9d66d from peer 078f8eff84c94dde9705ad1330922a11 ({username='slave'} at 127.10.145.4:43069)
I20240503 14:16:13.507967 28340 tablet_copy_service.cc:161] P acb93936f2f14e628eecb05d7d1f4d91: Beginning new tablet copy session on tablet 233ee592f6e843c8a68a5170e0f9d66d from peer 078f8eff84c94dde9705ad1330922a11 at {username='slave'} at 127.10.145.4:43069: session id = 078f8eff84c94dde9705ad1330922a11-233ee592f6e843c8a68a5170e0f9d66d
I20240503 14:16:13.511296 28340 tablet_copy_source_session.cc:205] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:13.511575 28186 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.518520 300 ts_tablet_manager.cc:1875] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:13.529773 28319 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.530762 405 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:13.531245 405 tablet_bootstrap.cc:492] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:16:13.531574 405 ts_tablet_manager.cc:1367] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.119s user 0.062s sys 0.000s
I20240503 14:16:13.532379 405 raft_consensus.cc:348] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:13.532722 405 raft_consensus.cc:729] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:16:13.532964 405 consensus_queue.cc:260] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:13.538591 371 tablet_copy_client.cc:523] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:13.551025 371 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:16:13.553874 405 ts_tablet_manager.cc:1398] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240503 14:16:13.548018 27661 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:13.555765 408 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:13.556205 408 tablet_bootstrap.cc:492] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268: Bootstrap complete.
I20240503 14:16:13.556538 408 ts_tablet_manager.cc:1367] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268: Time spent bootstrapping tablet: real 0.125s user 0.051s sys 0.010s
I20240503 14:16:13.557796 300 ts_tablet_manager.cc:1888] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 14:16:13.558897 28340 tablet_copy_service.cc:342] P acb93936f2f14e628eecb05d7d1f4d91: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-add941da26824cdda7489b5da990e879 received from {username='slave'} at 127.10.145.8:54229
I20240503 14:16:13.559028 28340 tablet_copy_service.cc:433] P acb93936f2f14e628eecb05d7d1f4d91: ending tablet copy session e938d875af584f1f88a2d4b50c589268-add941da26824cdda7489b5da990e879 on tablet add941da26824cdda7489b5da990e879 with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:16:13.561056 408 raft_consensus.cc:348] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:13.561447 408 raft_consensus.cc:729] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e938d875af584f1f88a2d4b50c589268, State: Initialized, Role: LEARNER
I20240503 14:16:13.561673 408 consensus_queue.cc:260] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
I20240503 14:16:13.562321 408 ts_tablet_manager.cc:1398] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:16:13.558516 300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 233ee592f6e843c8a68a5170e0f9d66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:13.563443 27682 tablet_copy_service.cc:342] P 7046cbd82d634976b53fe0cef5788f18: Request end of tablet copy session e938d875af584f1f88a2d4b50c589268-f15a6ff6acc242258172e585b1ea34c6 received from {username='slave'} at 127.10.145.8:47681
I20240503 14:16:13.563585 27682 tablet_copy_service.cc:433] P 7046cbd82d634976b53fe0cef5788f18: ending tablet copy session e938d875af584f1f88a2d4b50c589268-f15a6ff6acc242258172e585b1ea34c6 on tablet f15a6ff6acc242258172e585b1ea34c6 with peer e938d875af584f1f88a2d4b50c589268
I20240503 14:16:13.572904 300 tablet_copy_client.cc:783] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:13.573823 300 tablet_copy_client.cc:647] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:13.576699 300 tablet_copy_client.cc:523] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:13.584592 300 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:16:13.632226 28318 raft_consensus.cc:1203] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 14:16:13.720527 28057 consensus_queue.cc:237] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
W20240503 14:16:13.735312 27992 consensus_peers.cc:485] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 -> Peer acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): Couldn't send request to peer acb93936f2f14e628eecb05d7d1f4d91. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:13.735978 30302 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.737033 377 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.742236 28187 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.742655 30168 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.743062 377 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.743270 27794 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.745119 27922 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.746559 28448 raft_consensus.cc:1261] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:13.748586 32676 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.749231 32676 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.749775 32676 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.751473 372 consensus_queue.cc:1035] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:13.752288 371 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:13.752728 371 tablet_bootstrap.cc:492] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap complete.
I20240503 14:16:13.753015 371 ts_tablet_manager.cc:1367] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.202s user 0.068s sys 0.014s
I20240503 14:16:13.754734 371 raft_consensus.cc:348] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
I20240503 14:16:13.755080 371 raft_consensus.cc:729] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: LEARNER
I20240503 14:16:13.755326 371 consensus_queue.cc:260] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
I20240503 14:16:13.756101 371 ts_tablet_manager.cc:1398] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240503 14:16:13.757251 28471 tablet_copy_service.cc:342] P 9231e715a4e342a197d6c5456ba6b399: Request end of tablet copy session acb93936f2f14e628eecb05d7d1f4d91-48a05a802960425383984c379e238682 received from {username='slave'} at 127.10.145.6:40833
I20240503 14:16:13.757432 28471 tablet_copy_service.cc:433] P 9231e715a4e342a197d6c5456ba6b399: ending tablet copy session acb93936f2f14e628eecb05d7d1f4d91-48a05a802960425383984c379e238682 on tablet 48a05a802960425383984c379e238682 with peer acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:16:13.767939 300 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:13.768448 300 tablet_bootstrap.cc:492] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Bootstrap complete.
I20240503 14:16:13.768767 300 ts_tablet_manager.cc:1367] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.184s user 0.059s sys 0.013s
I20240503 14:16:13.769709 300 raft_consensus.cc:348] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:13.770058 300 raft_consensus.cc:729] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: LEARNER
I20240503 14:16:13.770388 300 consensus_queue.cc:260] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:13.771183 300 ts_tablet_manager.cc:1398] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:16:13.773659 28340 tablet_copy_service.cc:342] P acb93936f2f14e628eecb05d7d1f4d91: Request end of tablet copy session 078f8eff84c94dde9705ad1330922a11-233ee592f6e843c8a68a5170e0f9d66d received from {username='slave'} at 127.10.145.4:43069
I20240503 14:16:13.773799 28340 tablet_copy_service.cc:433] P acb93936f2f14e628eecb05d7d1f4d91: ending tablet copy session 078f8eff84c94dde9705ad1330922a11-233ee592f6e843c8a68a5170e0f9d66d on tablet 233ee592f6e843c8a68a5170e0f9d66d with peer 078f8eff84c94dde9705ad1330922a11
I20240503 14:16:13.775398 32676 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.779693 28448 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.780604 30168 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.785696 27922 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.789563 28187 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.790123 27521 catalog_manager.cc:5484] T 0e9e6daafc14437faff996cfb40f6dfb P 54dab61f1a204c0da4f56f98b3fd2690 reported cstate change: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.790769 27794 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.789563 30302 raft_consensus.cc:2932] T 0e9e6daafc14437faff996cfb40f6dfb P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } } }
I20240503 14:16:13.807698 300 ts_tablet_manager.cc:902] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Initiating tablet copy from peer a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119)
I20240503 14:16:13.825850 300 tablet_copy_client.cc:308] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.3:36119
I20240503 14:16:13.826725 27946 tablet_copy_service.cc:140] P a242f99f54b4494893064cfe3030b4ad: Received BeginTabletCopySession request for tablet 8ae969579ee545c9838d3c26e4abd66f from peer 078f8eff84c94dde9705ad1330922a11 ({username='slave'} at 127.10.145.4:51453)
I20240503 14:16:13.826900 27946 tablet_copy_service.cc:161] P a242f99f54b4494893064cfe3030b4ad: Beginning new tablet copy session on tablet 8ae969579ee545c9838d3c26e4abd66f from peer 078f8eff84c94dde9705ad1330922a11 at {username='slave'} at 127.10.145.4:51453: session id = 078f8eff84c94dde9705ad1330922a11-8ae969579ee545c9838d3c26e4abd66f
I20240503 14:16:13.830089 27946 tablet_copy_source_session.cc:205] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:13.831872 300 ts_tablet_manager.cc:1875] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:13.832543 371 ts_tablet_manager.cc:902] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Initiating tablet copy from peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:16:13.837960 496 raft_consensus.cc:1050] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb: attempting to promote NON_VOTER 078f8eff84c94dde9705ad1330922a11 to VOTER
I20240503 14:16:13.839465 300 ts_tablet_manager.cc:1888] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 14:16:13.839810 300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae969579ee545c9838d3c26e4abd66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:13.841130 496 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } }
I20240503 14:16:13.844153 30034 raft_consensus.cc:1203] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 14:16:13.845243 300 tablet_copy_client.cc:783] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:13.845561 300 tablet_copy_client.cc:647] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:13.850445 28448 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:13.851279 30034 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:13.851600 28057 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:13.851910 30302 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:13.852514 28187 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:13.853091 300 tablet_copy_client.cc:523] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:13.853812 27661 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:13.854053 28318 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:13.854511 32538 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:13.855232 32538 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:13.858317 32538 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20240503 14:16:13.860757 300 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Bootstrap starting.
I20240503 14:16:13.875490 30034 raft_consensus.cc:1203] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:13.880831 30284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:13.894224 32712 raft_consensus.cc:1050] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91: attempting to promote NON_VOTER 54dab61f1a204c0da4f56f98b3fd2690 to VOTER
I20240503 14:16:13.894834 28035 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:13.895470 32712 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:13.912632 30149 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:13.942966 496 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:13.944380 496 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240503 14:16:13.961117 30034 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:13.961510 30302 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:13.962412 32711 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:13.963145 32711 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:13.971328 371 tablet_copy_client.cc:308] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:45951
I20240503 14:16:13.973343 27922 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 14:16:13.969907 27730 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.8:44583, user_credentials={real_user=slave}} blocked reactor thread for 114170us
I20240503 14:16:13.979629 507 raft_consensus.cc:1050] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422: attempting to promote NON_VOTER 7046cbd82d634976b53fe0cef5788f18 to VOTER
I20240503 14:16:13.974985 32562 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:13.982273 30168 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 LEARNER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:13.984851 32711 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:13.990032 28077 tablet_copy_service.cc:140] P 078f8eff84c94dde9705ad1330922a11: Received BeginTabletCopySession request for tablet 9f00380931064f71bc78626af72488f3 from peer acb93936f2f14e628eecb05d7d1f4d91 ({username='slave'} at 127.10.145.6:60381)
I20240503 14:16:13.990232 28077 tablet_copy_service.cc:161] P 078f8eff84c94dde9705ad1330922a11: Beginning new tablet copy session on tablet 9f00380931064f71bc78626af72488f3 from peer acb93936f2f14e628eecb05d7d1f4d91 at {username='slave'} at 127.10.145.6:60381: session id = acb93936f2f14e628eecb05d7d1f4d91-9f00380931064f71bc78626af72488f3
I20240503 14:16:13.993320 28077 tablet_copy_source_session.cc:205] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:13.995267 28057 raft_consensus.cc:1203] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 14:16:13.979920 27729 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.10.145.1:35261, user_credentials={real_user=slave}} blocked reactor thread for 125449us
I20240503 14:16:14.024909 32538 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.027674 27794 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.031558 371 ts_tablet_manager.cc:1875] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:14.038002 300 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:14.038563 300 tablet_bootstrap.cc:492] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Bootstrap complete.
I20240503 14:16:14.038914 300 ts_tablet_manager.cc:1367] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Time spent bootstrapping tablet: real 0.178s user 0.054s sys 0.000s
I20240503 14:16:14.043797 518 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.046504 507 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } }
I20240503 14:16:14.039829 300 raft_consensus.cc:348] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:14.048921 300 raft_consensus.cc:729] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 078f8eff84c94dde9705ad1330922a11, State: Initialized, Role: LEARNER
I20240503 14:16:14.050470 32711 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.049232 300 consensus_queue.cc:260] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:14.055756 300 ts_tablet_manager.cc:1398] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240503 14:16:14.056493 27946 tablet_copy_service.cc:342] P a242f99f54b4494893064cfe3030b4ad: Request end of tablet copy session 078f8eff84c94dde9705ad1330922a11-8ae969579ee545c9838d3c26e4abd66f received from {username='slave'} at 127.10.145.4:51453
I20240503 14:16:14.056663 27946 tablet_copy_service.cc:433] P a242f99f54b4494893064cfe3030b4ad: ending tablet copy session 078f8eff84c94dde9705ad1330922a11-8ae969579ee545c9838d3c26e4abd66f on tablet 8ae969579ee545c9838d3c26e4abd66f with peer 078f8eff84c94dde9705ad1330922a11
I20240503 14:16:14.071086 28448 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.072382 32562 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.073086 32562 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.081470 27922 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.081447 30034 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.086771 30168 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.087615 27520 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 11 to 12, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.092860 496 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.097079 28186 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.103209 32711 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.102797 30034 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.104684 30304 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.110249 27515 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 reported cstate change: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.085835 30302 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.126080 28057 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.136808 28448 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.139650 27793 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.180344 28430 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:14.186870 371 ts_tablet_manager.cc:1888] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 14:16:14.187409 371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f00380931064f71bc78626af72488f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:14.188277 28186 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.191262 28449 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.193707 30304 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.194420 27659 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEARNER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.194486 514 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.196961 525 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.209861 27904 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:14.212667 28318 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.221771 30168 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.221858 27922 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.222471 514 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.223901 525 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.227103 28318 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.227710 525 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.230252 28057 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.231123 507 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.238474 27793 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.240223 507 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.244746 371 tablet_copy_client.cc:783] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:14.250231 28055 raft_consensus.cc:1203] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:14.250993 558 raft_consensus.cc:1050] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad: attempting to promote NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 to VOTER
I20240503 14:16:14.264041 28318 raft_consensus.cc:1203] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:16:14.265172 27642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:14.272717 371 tablet_copy_client.cc:647] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:14.276351 520 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.277926 27794 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.283665 28186 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.284967 27922 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.284736 27661 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.289673 28057 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.294561 371 tablet_copy_client.cc:523] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:14.292541 30304 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.297660 331 raft_consensus.cc:1050] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:16:14.297309 27520 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 11 to 12, 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.298861 331 consensus_queue.cc:237] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:14.305222 371 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:16:14.315369 27661 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.322616 27793 consensus_queue.cc:237] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } }
I20240503 14:16:14.325480 27772 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:14.324182 28319 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.332549 30302 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.331624 558 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } }
I20240503 14:16:14.333216 506 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.333845 28057 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.334627 30034 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.337509 556 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.338232 506 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.345567 506 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.346354 28057 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.348472 28299 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:14.348927 28187 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.349622 496 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.351130 28187 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.352695 30034 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.353626 32675 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.354172 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d84acfe4f36f4193970fd2230229a41f with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:14.355144 30302 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.357462 28448 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.357600 30032 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.358655 32675 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.364950 27513 catalog_manager.cc:5484] T d84acfe4f36f4193970fd2230229a41f P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.370433 28317 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.371027 28186 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.371240 27922 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.371194 338 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.371856 338 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.372726 28430 tablet_service.cc:1514] Processing DeleteTablet for tablet d84acfe4f36f4193970fd2230229a41f with delete_type TABLET_DATA_TOMBSTONED (TS 9231e715a4e342a197d6c5456ba6b399 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:60466
I20240503 14:16:14.374601 28319 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.375154 27661 raft_consensus.cc:1261] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.377521 556 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.381425 526 raft_consensus.cc:1050] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91: attempting to promote NON_VOTER e938d875af584f1f88a2d4b50c589268 to VOTER
I20240503 14:16:14.381665 28319 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.382728 526 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:14.390170 30168 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.401592 28168 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:14.402585 28186 consensus_queue.cc:237] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } }
I20240503 14:16:14.412432 30169 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.416121 27794 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.416096 496 consensus_queue.cc:1035] T d84acfe4f36f4193970fd2230229a41f P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.419245 338 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.421314 338 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.422430 338 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.427065 545 raft_consensus.cc:1050] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91: attempting to promote NON_VOTER 078f8eff84c94dde9705ad1330922a11 to VOTER
I20240503 14:16:14.427493 30302 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.428433 520 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.428659 28319 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.428436 545 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } }
I20240503 14:16:14.429334 520 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.429255 27659 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.430333 507 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.433652 28057 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.433692 27793 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.434759 516 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.435536 507 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.435966 507 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.437582 28057 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.439621 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12dae8a392374b91be378ec431838686 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:14.441143 27792 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.444784 27661 raft_consensus.cc:2932] T d84acfe4f36f4193970fd2230229a41f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.445349 27659 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.447212 575 tablet_replica.cc:330] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399: stopping tablet replica
I20240503 14:16:14.447484 575 raft_consensus.cc:2229] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:14.448045 575 raft_consensus.cc:2258] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:14.449996 28317 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.451539 27515 catalog_manager.cc:5484] T 12dae8a392374b91be378ec431838686 P 078f8eff84c94dde9705ad1330922a11 reported cstate change: config changed from index 12 to 13, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New cstate: current_term: 1 leader_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.457039 570 ts_tablet_manager.cc:902] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Initiating tablet copy from peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951)
I20240503 14:16:14.460788 27904 tablet_service.cc:1514] Processing DeleteTablet for tablet 12dae8a392374b91be378ec431838686 with delete_type TABLET_DATA_TOMBSTONED (TS a242f99f54b4494893064cfe3030b4ad not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:46680
I20240503 14:16:14.462215 338 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.466099 575 ts_tablet_manager.cc:1875] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:14.469347 28318 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.469540 27794 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.470582 27515 catalog_manager.cc:5484] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:16:14.472456 30302 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.474086 30032 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.477979 28186 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.489964 30034 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268 [term 1 LEARNER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.491786 30016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:14.498618 30165 raft_consensus.cc:1261] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer c91f285b8f4f4a27bcdd0ada303c1422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.504217 28448 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.509972 525 consensus_queue.cc:1035] T 12dae8a392374b91be378ec431838686 P c91f285b8f4f4a27bcdd0ada303c1422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.514008 30168 raft_consensus.cc:2932] T 12dae8a392374b91be378ec431838686 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a242f99f54b4494893064cfe3030b4ad (127.10.145.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } attrs { promote: false } } }
I20240503 14:16:14.515587 575 ts_tablet_manager.cc:1888] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 14:16:14.516069 32562 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.516533 30302 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.519223 32562 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.521015 575 log.cc:1196] T d84acfe4f36f4193970fd2230229a41f P 9231e715a4e342a197d6c5456ba6b399: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/wals/d84acfe4f36f4193970fd2230229a41f
I20240503 14:16:14.522449 27506 catalog_manager.cc:4830] TS 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): tablet d84acfe4f36f4193970fd2230229a41f (table rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]) successfully deleted
I20240503 14:16:14.528456 27924 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.529245 32711 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.541055 30165 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.541697 27794 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.542244 32562 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.542618 32711 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.542742 28449 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.546083 32711 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.548116 28186 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.548756 526 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.564296 590 tablet_replica.cc:330] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad: stopping tablet replica
I20240503 14:16:14.564350 30302 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.564682 590 raft_consensus.cc:2229] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:14.565047 561 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.565455 590 raft_consensus.cc:2258] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:14.568524 30167 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.570420 561 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.574539 590 ts_tablet_manager.cc:1875] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:14.577344 28318 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.578114 454 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.578356 526 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.581228 27924 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.591831 30302 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.598806 28450 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.599557 454 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240503 14:16:14.599699 27515 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 reported cstate change: config changed from index 11 to 12, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.611340 28057 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.611115 27794 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.612190 454 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.617846 28186 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.624962 27661 consensus_queue.cc:237] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:14.639230 28055 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.640101 32711 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.640739 28186 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.641531 454 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.645581 27659 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.646144 590 ts_tablet_manager.cc:1888] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 14:16:14.646289 590 log.cc:1196] T 12dae8a392374b91be378ec431838686 P a242f99f54b4494893064cfe3030b4ad: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-2/wal/wals/12dae8a392374b91be378ec431838686
I20240503 14:16:14.646430 561 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.647179 27508 catalog_manager.cc:4830] TS a242f99f54b4494893064cfe3030b4ad (127.10.145.3:36119): tablet 12dae8a392374b91be378ec431838686 (table rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]) successfully deleted
I20240503 14:16:14.648458 30032 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:14.649292 574 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:14.653923 28317 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:14.654728 28188 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:14.655108 27926 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:14.655380 32675 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:14.655763 338 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:14.656224 32675 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.656217 27922 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.659001 27794 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:14.659689 574 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:14.660923 32675 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.663233 27794 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.665196 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f15a6ff6acc242258172e585b1ea34c6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:14.666591 30032 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.666574 28317 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.666663 30033 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.667513 28188 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.667021 28187 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.675755 27520 catalog_manager.cc:5484] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 reported cstate change: config changed from index 9 to 10, VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) evicted. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.676312 30166 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.677249 595 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.677999 595 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.678542 595 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.682557 27924 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.684432 32711 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.697701 28317 consensus_queue.cc:237] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } }
I20240503 14:16:14.701238 30302 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.702111 32711 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.702461 27661 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.703295 32711 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.710011 454 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:14.716565 30302 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:14.725690 30164 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.725836 30033 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.726649 27518 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 reported cstate change: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:14.732146 28057 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:14.733940 27659 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:14.737556 28318 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:14.747462 28446 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e938d875af584f1f88a2d4b50c589268 (127.10.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.762679 28446 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:14.767822 28444 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:14.767853 28319 consensus_queue.cc:237] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } }
I20240503 14:16:14.768968 574 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:14.764825 28449 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:14.771025 595 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:14.771581 32562 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.772635 28055 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.725100 30165 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:14.776968 28185 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.783675 27923 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.783883 30164 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.783504 27513 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.784688 595 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.785017 30034 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.785257 27794 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.785403 32712 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.786661 28444 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.787263 32712 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.787807 32712 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.787796 30033 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.790354 32712 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.793642 32712 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.794368 27794 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.795058 28448 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.801689 28185 raft_consensus.cc:1261] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.801581 27512 catalog_manager.cc:5484] T add941da26824cdda7489b5da990e879 P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 12 to 13, VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) evicted. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.802145 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet add941da26824cdda7489b5da990e879 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:14.803246 32711 consensus_queue.cc:1035] T add941da26824cdda7489b5da990e879 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.804721 27923 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.814044 30168 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.818135 28185 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.832139 568 raft_consensus.cc:1050] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad: attempting to promote NON_VOTER 078f8eff84c94dde9705ad1330922a11 to VOTER
I20240503 14:16:14.833410 568 consensus_queue.cc:237] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } }
I20240503 14:16:14.834968 27659 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.840130 28444 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.840409 28449 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.851675 371 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:14.852167 371 tablet_bootstrap.cc:492] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap complete.
I20240503 14:16:14.852531 371 ts_tablet_manager.cc:1367] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.547s user 0.054s sys 0.006s
I20240503 14:16:14.827749 27924 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:14.871905 371 raft_consensus.cc:348] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
I20240503 14:16:14.872247 371 raft_consensus.cc:729] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: LEARNER
I20240503 14:16:14.872514 371 consensus_queue.cc:260] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
I20240503 14:16:14.873226 371 ts_tablet_manager.cc:1398] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20240503 14:16:14.879520 28077 tablet_copy_service.cc:342] P 078f8eff84c94dde9705ad1330922a11: Request end of tablet copy session acb93936f2f14e628eecb05d7d1f4d91-9f00380931064f71bc78626af72488f3 received from {username='slave'} at 127.10.145.6:60381
I20240503 14:16:14.879710 28077 tablet_copy_service.cc:433] P 078f8eff84c94dde9705ad1330922a11: ending tablet copy session acb93936f2f14e628eecb05d7d1f4d91-9f00380931064f71bc78626af72488f3 on tablet 9f00380931064f71bc78626af72488f3 with peer acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:16:14.890517 467 raft_consensus.cc:1050] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399: attempting to promote NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 to VOTER
I20240503 14:16:14.892204 467 consensus_queue.cc:237] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } }
I20240503 14:16:14.891397 28185 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:14.899021 27794 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.900255 595 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.904057 570 tablet_copy_client.cc:308] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.4:45951
I20240503 14:16:14.904070 30165 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.904685 526 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.905453 28077 tablet_copy_service.cc:140] P 078f8eff84c94dde9705ad1330922a11: Received BeginTabletCopySession request for tablet 0e9e6daafc14437faff996cfb40f6dfb from peer acb93936f2f14e628eecb05d7d1f4d91 ({username='slave'} at 127.10.145.6:60381)
I20240503 14:16:14.905668 28077 tablet_copy_service.cc:161] P 078f8eff84c94dde9705ad1330922a11: Beginning new tablet copy session on tablet 0e9e6daafc14437faff996cfb40f6dfb from peer acb93936f2f14e628eecb05d7d1f4d91 at {username='slave'} at 127.10.145.6:60381: session id = acb93936f2f14e628eecb05d7d1f4d91-0e9e6daafc14437faff996cfb40f6dfb
I20240503 14:16:14.908648 28077 tablet_copy_source_session.cc:205] T 0e9e6daafc14437faff996cfb40f6dfb P 078f8eff84c94dde9705ad1330922a11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:14.910108 28446 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.911317 570 ts_tablet_manager.cc:1875] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:14.911908 526 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.914831 595 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.920210 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c23af4dc19c412685927c70cbee7124 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:14.922422 30165 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.927868 28185 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.928753 32712 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.929749 30034 raft_consensus.cc:2932] T add941da26824cdda7489b5da990e879 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:14.931152 28185 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.939549 28185 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.939189 30302 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.939668 27794 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.940219 625 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 14:16:14.941869 28057 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.942485 625 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.943691 28317 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.944515 625 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.944758 30034 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.945510 322 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.949072 27515 catalog_manager.cc:5484] T 4c23af4dc19c412685927c70cbee7124 P 01173ed91fd341e3a4a4599118167e52 reported cstate change: config changed from index 12 to 13, VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) evicted. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.968346 27661 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.969488 625 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.969996 27794 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.970661 322 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:14.991104 27926 raft_consensus.cc:1261] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:14.993084 526 consensus_queue.cc:1035] T 4c23af4dc19c412685927c70cbee7124 P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:14.996532 27925 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:14.998584 27923 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } }
I20240503 14:16:15.003386 570 ts_tablet_manager.cc:1888] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 14:16:15.003715 570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9e6daafc14437faff996cfb40f6dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:15.003207 27922 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:14.985594 28446 raft_consensus.cc:2932] T 4c23af4dc19c412685927c70cbee7124 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } }
I20240503 14:16:15.017890 624 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:15.027979 28057 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 LEARNER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:15.028534 558 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:15.029119 27793 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:15.030390 28449 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:15.031332 558 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:15.031677 454 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:15.032424 27661 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:15.033202 454 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:15.039860 28186 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:15.040467 28319 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:15.040481 568 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:15.041134 558 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:15.045571 631 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.048877 30034 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.056591 27924 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.066641 28318 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.076452 27518 catalog_manager.cc:5484] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 reported cstate change: config changed from index 8 to 9, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:16:15.073398 28057 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.084455 27792 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.089951 30033 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:15.082729 568 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.099624 570 tablet_copy_client.cc:783] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:15.099933 570 tablet_copy_client.cc:647] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:15.102499 570 tablet_copy_client.cc:523] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:15.079993 27659 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.083933 27922 consensus_queue.cc:237] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } }
I20240503 14:16:15.090637 561 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:15.113809 28319 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.101063 27520 catalog_manager.cc:5484] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb reported cstate change: config changed from index 8 to 9, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.117712 30167 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.124585 28185 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.133230 28449 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.138393 28185 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.138393 28057 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.139604 27661 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.154465 30149 tablet_service.cc:1514] Processing DeleteTablet for tablet f15a6ff6acc242258172e585b1ea34c6 with delete_type TABLET_DATA_TOMBSTONED (TS 54dab61f1a204c0da4f56f98b3fd2690 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33130
I20240503 14:16:15.155908 30016 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c23af4dc19c412685927c70cbee7124 with delete_type TABLET_DATA_TOMBSTONED (TS e938d875af584f1f88a2d4b50c589268 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:52632
I20240503 14:16:15.173033 27794 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.175720 654 tablet_replica.cc:330] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: stopping tablet replica
I20240503 14:16:15.184466 28319 consensus_queue.cc:237] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } }
I20240503 14:16:15.186296 654 raft_consensus.cc:2229] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:15.186821 654 pending_rounds.cc:62] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: Trying to abort 1 pending ops.
I20240503 14:16:15.187009 654 pending_rounds.cc:68] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7023598691524800512 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f15a6ff6acc242258172e585b1ea34c6" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } } }
I20240503 14:16:15.187888 654 raft_consensus.cc:2894] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20240503 14:16:15.190137 30034 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.195457 30033 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.196452 28185 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.196653 27926 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.196537 595 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.197463 28057 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.197531 595 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.198066 595 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.198302 32712 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.202960 32712 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.203784 570 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap starting.
I20240503 14:16:15.204217 28185 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.204753 28055 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.205613 30302 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.206333 651 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.208215 27507 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 233ee592f6e843c8a68a5170e0f9d66d with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:15.210490 28319 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.210740 30284 tablet_service.cc:1514] Processing DeleteTablet for tablet add941da26824cdda7489b5da990e879 with delete_type TABLET_DATA_TOMBSTONED (TS 01173ed91fd341e3a4a4599118167e52 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:58692
I20240503 14:16:15.210902 27659 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.210739 27519 catalog_manager.cc:5484] T 233ee592f6e843c8a68a5170e0f9d66d P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New cstate: current_term: 1 leader_uuid: "acb93936f2f14e628eecb05d7d1f4d91" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.211702 545 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.213693 27659 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.217832 28430 tablet_service.cc:1514] Processing DeleteTablet for tablet 233ee592f6e843c8a68a5170e0f9d66d with delete_type TABLET_DATA_TOMBSTONED (TS 9231e715a4e342a197d6c5456ba6b399 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:60466
I20240503 14:16:15.221890 30033 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.222191 27926 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.210289 30167 raft_consensus.cc:1261] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer acb93936f2f14e628eecb05d7d1f4d91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.224563 654 raft_consensus.cc:2866] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:16:15.225813 28055 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.225796 545 consensus_queue.cc:1035] T 233ee592f6e843c8a68a5170e0f9d66d P acb93936f2f14e628eecb05d7d1f4d91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.226701 568 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.229449 561 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.233101 27661 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.233961 651 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.234309 30167 raft_consensus.cc:2932] T 233ee592f6e843c8a68a5170e0f9d66d P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.236680 561 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.237833 30165 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.238972 27662 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.239236 28319 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.240048 654 raft_consensus.cc:2258] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:15.243602 651 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.246822 660 tablet_replica.cc:330] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: stopping tablet replica
I20240503 14:16:15.247663 660 raft_consensus.cc:2229] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:15.248275 660 raft_consensus.cc:2258] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:15.249523 30165 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.250263 28187 raft_consensus.cc:1261] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:15.250996 650 consensus_queue.cc:1035] T e895331729d3400384f20aa3895540de P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:15.252004 661 tablet_replica.cc:330] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399: stopping tablet replica
I20240503 14:16:15.253161 661 raft_consensus.cc:2229] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:15.254122 661 raft_consensus.cc:2258] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:15.254127 28185 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.254751 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e895331729d3400384f20aa3895540de with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:15.257205 661 ts_tablet_manager.cc:1875] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.256959 27518 catalog_manager.cc:5484] T e895331729d3400384f20aa3895540de P 54dab61f1a204c0da4f56f98b3fd2690 reported cstate change: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.258790 30302 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.264237 28430 tablet_service.cc:1514] Processing DeleteTablet for tablet e895331729d3400384f20aa3895540de with delete_type TABLET_DATA_TOMBSTONED (TS 9231e715a4e342a197d6c5456ba6b399 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:60466
I20240503 14:16:15.266896 654 ts_tablet_manager.cc:1875] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.266104 28055 raft_consensus.cc:2932] T e895331729d3400384f20aa3895540de P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.283471 28449 consensus_queue.cc:237] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } }
I20240503 14:16:15.288403 28319 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.289180 631 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.294791 27794 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.295480 624 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.295910 27659 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.295907 28185 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.296504 624 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.297220 27924 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.298147 625 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.298748 467 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.300662 631 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.301827 27924 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.302949 28185 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.304214 27662 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.306178 27506 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48a05a802960425383984c379e238682 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:15.307461 28318 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.309913 27520 catalog_manager.cc:5484] T 48a05a802960425383984c379e238682 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 9 to 10, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.315742 27794 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.316861 28035 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a05a802960425383984c379e238682 with delete_type TABLET_DATA_TOMBSTONED (TS 078f8eff84c94dde9705ad1330922a11 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35710
I20240503 14:16:15.329475 661 ts_tablet_manager.cc:1888] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 14:16:15.330101 654 ts_tablet_manager.cc:1888] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:15.330389 654 log.cc:1196] T f15a6ff6acc242258172e585b1ea34c6 P 54dab61f1a204c0da4f56f98b3fd2690: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-8/wal/wals/f15a6ff6acc242258172e585b1ea34c6
I20240503 14:16:15.330583 661 log.cc:1196] T 233ee592f6e843c8a68a5170e0f9d66d P 9231e715a4e342a197d6c5456ba6b399: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/wals/233ee592f6e843c8a68a5170e0f9d66d
I20240503 14:16:15.332566 27508 catalog_manager.cc:4830] TS 54dab61f1a204c0da4f56f98b3fd2690 (127.10.145.9:39057): tablet f15a6ff6acc242258172e585b1ea34c6 (table rebalance_test_table_1 [id=3551dc0e1c744a43b61007e6b7f17d97]) successfully deleted
I20240503 14:16:15.332640 30033 raft_consensus.cc:1261] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.333923 631 consensus_queue.cc:1035] T 48a05a802960425383984c379e238682 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.335738 30033 raft_consensus.cc:2932] T 48a05a802960425383984c379e238682 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.338346 661 tablet_replica.cc:330] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399: stopping tablet replica
I20240503 14:16:15.338524 27506 catalog_manager.cc:4830] TS 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): tablet 233ee592f6e843c8a68a5170e0f9d66d (table rebalance_test_table_2 [id=6c209ad732404a06889d91432f7c230d]) successfully deleted
I20240503 14:16:15.338590 661 raft_consensus.cc:2229] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:15.338891 661 raft_consensus.cc:2258] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:15.341931 663 tablet_replica.cc:330] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11: stopping tablet replica
I20240503 14:16:15.342214 663 raft_consensus.cc:2229] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:15.342738 663 raft_consensus.cc:2258] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:15.344247 663 ts_tablet_manager.cc:1875] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.344926 655 tablet_replica.cc:330] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: stopping tablet replica
I20240503 14:16:15.345633 655 raft_consensus.cc:2229] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:15.346222 655 raft_consensus.cc:2258] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:15.348062 655 ts_tablet_manager.cc:1875] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.351627 663 ts_tablet_manager.cc:1888] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:15.351862 663 log.cc:1196] T 48a05a802960425383984c379e238682 P 078f8eff84c94dde9705ad1330922a11: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-3/wal/wals/48a05a802960425383984c379e238682
I20240503 14:16:15.353551 27508 catalog_manager.cc:4830] TS 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): tablet 48a05a802960425383984c379e238682 (table rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]) successfully deleted
I20240503 14:16:15.356601 660 ts_tablet_manager.cc:1875] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.358870 28319 raft_consensus.cc:1203] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 14:16:15.378764 661 ts_tablet_manager.cc:1875] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.384045 661 ts_tablet_manager.cc:1888] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 14:16:15.393296 661 log.cc:1196] T e895331729d3400384f20aa3895540de P 9231e715a4e342a197d6c5456ba6b399: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-6/wal/wals/e895331729d3400384f20aa3895540de
I20240503 14:16:15.394578 27506 catalog_manager.cc:4830] TS 9231e715a4e342a197d6c5456ba6b399 (127.10.145.7:43675): tablet e895331729d3400384f20aa3895540de (table rebalance_test_table_5 [id=daab67e27881420895980ab143f7f07d]) successfully deleted
I20240503 14:16:15.419639 655 ts_tablet_manager.cc:1888] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 14:16:15.419986 655 log.cc:1196] T 4c23af4dc19c412685927c70cbee7124 P e938d875af584f1f88a2d4b50c589268: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-7/wal/wals/4c23af4dc19c412685927c70cbee7124
I20240503 14:16:15.421311 27508 catalog_manager.cc:4830] TS e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): tablet 4c23af4dc19c412685927c70cbee7124 (table rebalance_test_table_4 [id=f9deba14e1804f2088da718c33600cc2]) successfully deleted
I20240503 14:16:15.424897 660 ts_tablet_manager.cc:1888] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240503 14:16:15.425112 660 log.cc:1196] T add941da26824cdda7489b5da990e879 P 01173ed91fd341e3a4a4599118167e52: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-9/wal/wals/add941da26824cdda7489b5da990e879
I20240503 14:16:15.426162 27508 catalog_manager.cc:4830] TS 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): tablet add941da26824cdda7489b5da990e879 (table rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]) successfully deleted
I20240503 14:16:15.437104 27659 consensus_queue.cc:237] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } }
I20240503 14:16:15.450592 30033 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.451516 614 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:15.469689 27598 consensus_peers.cc:485] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 -> Peer 01173ed91fd341e3a4a4599118167e52 (127.10.145.10:40989): Couldn't send request to peer 01173ed91fd341e3a4a4599118167e52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbb645fc3c5c422b9e3fabba8ce12873. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:15.471391 27924 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.472375 27794 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.473052 338 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.475116 614 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.475236 28449 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.475977 613 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.476763 28185 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.478005 614 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.531003 27926 consensus_queue.cc:237] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } }
I20240503 14:16:15.534317 28055 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.534998 561 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.540802 27662 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.540875 30034 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.541623 561 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.542907 561 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.546042 28446 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.546741 650 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.549624 568 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.550227 613 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:15.554695 28446 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.560436 27922 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:15.564990 27793 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.566447 651 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.569262 27515 catalog_manager.cc:5484] T bbb645fc3c5c422b9e3fabba8ce12873 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:15.570825 27794 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:15.568194 28185 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:15.571508 30034 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.576422 28319 raft_consensus.cc:1261] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.579246 27513 catalog_manager.cc:5484] T 8ae969579ee545c9838d3c26e4abd66f P e938d875af584f1f88a2d4b50c589268 reported cstate change: config changed from index 9 to 10, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New cstate: current_term: 1 leader_uuid: "a242f99f54b4494893064cfe3030b4ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.581657 338 consensus_queue.cc:1035] T bbb645fc3c5c422b9e3fabba8ce12873 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.584555 28299 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae969579ee545c9838d3c26e4abd66f with delete_type TABLET_DATA_TOMBSTONED (TS acb93936f2f14e628eecb05d7d1f4d91 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39840
I20240503 14:16:15.584753 27794 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.591746 690 tablet_replica.cc:330] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91: stopping tablet replica
I20240503 14:16:15.592080 690 raft_consensus.cc:2229] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:15.592542 690 raft_consensus.cc:2258] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:15.593145 28449 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:15.598342 690 ts_tablet_manager.cc:1875] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.605082 28186 raft_consensus.cc:1261] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer a242f99f54b4494893064cfe3030b4ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:15.605896 561 consensus_queue.cc:1035] T 8ae969579ee545c9838d3c26e4abd66f P a242f99f54b4494893064cfe3030b4ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:15.607797 28186 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.610567 27508 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ae969579ee545c9838d3c26e4abd66f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:16:15.613171 28055 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.616596 27659 raft_consensus.cc:2932] T 8ae969579ee545c9838d3c26e4abd66f P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: false } } }
I20240503 14:16:15.620862 690 ts_tablet_manager.cc:1888] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:15.621049 690 log.cc:1196] T 8ae969579ee545c9838d3c26e4abd66f P acb93936f2f14e628eecb05d7d1f4d91: Deleting WAL directory at /tmp/dist-test-taskFZBpI6/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714745515850677-10820-0/raft_consensus-itest-cluster/ts-5/wal/wals/8ae969579ee545c9838d3c26e4abd66f
I20240503 14:16:15.626052 27507 catalog_manager.cc:4830] TS acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6:36073): tablet 8ae969579ee545c9838d3c26e4abd66f (table rebalance_test_table_0 [id=8991da2b01db4e0f832fc7bff8693981]) successfully deleted
I20240503 14:16:15.650424 30033 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:15.655562 28319 raft_consensus.cc:2932] T bbb645fc3c5c422b9e3fabba8ce12873 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 01173ed91fd341e3a4a4599118167e52 (127.10.145.10) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: NON_VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: true } } }
I20240503 14:16:15.659209 570 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:15.659749 570 tablet_bootstrap.cc:492] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Bootstrap complete.
I20240503 14:16:15.660073 570 ts_tablet_manager.cc:1367] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Time spent bootstrapping tablet: real 0.456s user 0.056s sys 0.000s
I20240503 14:16:15.660913 570 raft_consensus.cc:348] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
I20240503 14:16:15.661305 570 raft_consensus.cc:729] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: acb93936f2f14e628eecb05d7d1f4d91, State: Initialized, Role: LEARNER
I20240503 14:16:15.661588 570 consensus_queue.cc:260] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: NON_VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: true } }
I20240503 14:16:15.662309 570 ts_tablet_manager.cc:1398] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:15.682320 28077 tablet_copy_service.cc:342] P 078f8eff84c94dde9705ad1330922a11: Request end of tablet copy session acb93936f2f14e628eecb05d7d1f4d91-0e9e6daafc14437faff996cfb40f6dfb received from {username='slave'} at 127.10.145.6:60381
I20240503 14:16:15.682492 28077 tablet_copy_service.cc:433] P 078f8eff84c94dde9705ad1330922a11: ending tablet copy session acb93936f2f14e628eecb05d7d1f4d91-0e9e6daafc14437faff996cfb40f6dfb on tablet 0e9e6daafc14437faff996cfb40f6dfb with peer acb93936f2f14e628eecb05d7d1f4d91
I20240503 14:16:15.696347 28319 raft_consensus.cc:1203] T 0e9e6daafc14437faff996cfb40f6dfb P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 14:16:15.718667 28446 consensus_queue.cc:237] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } }
W20240503 14:16:15.728763 28387 consensus_peers.cc:485] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 -> Peer e938d875af584f1f88a2d4b50c589268 (127.10.145.8:44583): Couldn't send request to peer e938d875af584f1f88a2d4b50c589268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44df52bbf3534802bf7217fb5cdb4326. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:15.728865 28055 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.729629 467 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.729935 30165 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.731036 27926 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.731500 28186 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.731849 27659 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.732214 625 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.732586 27794 raft_consensus.cc:1261] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 9231e715a4e342a197d6c5456ba6b399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.732561 467 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.732887 631 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.733209 624 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.733775 322 consensus_queue.cc:1035] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.751852 467 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 9231e715a4e342a197d6c5456ba6b399 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:15.754105 27659 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:15.757647 30165 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:15.761343 27926 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:15.772084 27515 catalog_manager.cc:5484] T 44df52bbf3534802bf7217fb5cdb4326 P 7046cbd82d634976b53fe0cef5788f18 reported cstate change: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New cstate: current_term: 1 leader_uuid: "9231e715a4e342a197d6c5456ba6b399" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:15.772910 28186 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:15.775693 28055 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 078f8eff84c94dde9705ad1330922a11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:15.776255 27794 raft_consensus.cc:2932] T 44df52bbf3534802bf7217fb5cdb4326 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e938d875af584f1f88a2d4b50c589268 (127.10.145.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: NON_VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: true } } }
I20240503 14:16:15.782420 27659 consensus_queue.cc:237] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:15.790083 28446 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 14:16:15.790215 27598 consensus_peers.cc:485] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 -> Peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): Couldn't send request to peer 078f8eff84c94dde9705ad1330922a11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:15.790549 27794 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:15.791108 27926 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:15.791115 28186 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:15.791388 28319 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:15.792217 32675 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:15.792758 684 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:15.793072 30033 raft_consensus.cc:1261] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 7046cbd82d634976b53fe0cef5788f18: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:15.793546 614 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:15.794039 614 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:15.794538 614 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:15.798393 32675 consensus_queue.cc:1035] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:16:15.804673 32675 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 7046cbd82d634976b53fe0cef5788f18 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.806533 27794 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.807333 28186 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.810729 28446 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.813822 27515 catalog_manager.cc:5484] T f15a6ff6acc242258172e585b1ea34c6 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New cstate: current_term: 1 leader_uuid: "7046cbd82d634976b53fe0cef5788f18" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.823786 27926 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.828011 30033 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.842159 28319 raft_consensus.cc:2932] T f15a6ff6acc242258172e585b1ea34c6 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.855813 27794 consensus_queue.cc:237] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } }
I20240503 14:16:15.859836 30033 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:16:15.860334 27730 consensus_peers.cc:485] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb -> Peer 078f8eff84c94dde9705ad1330922a11 (127.10.145.4:45951): Couldn't send request to peer 078f8eff84c94dde9705ad1330922a11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:15.861258 28446 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.861351 556 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.862556 556 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.865021 28186 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.865021 28319 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.865103 27659 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.865695 27926 raft_consensus.cc:1261] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 74318b6270b64efaa79867167fc08bbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:15.865765 518 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.866453 518 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.866863 556 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.877485 738 consensus_queue.cc:1035] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:15.882961 556 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 74318b6270b64efaa79867167fc08bbb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.883940 28186 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.888087 28446 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.889035 27926 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.894260 27515 catalog_manager.cc:5484] T 06b7e707cec54bc8bc5e649c3b37dfb5 P c91f285b8f4f4a27bcdd0ada303c1422 reported cstate change: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New cstate: current_term: 1 leader_uuid: "74318b6270b64efaa79867167fc08bbb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.891656 30033 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P e938d875af584f1f88a2d4b50c589268 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.917459 27662 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P 7046cbd82d634976b53fe0cef5788f18 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.919250 28319 raft_consensus.cc:2932] T 06b7e707cec54bc8bc5e649c3b37dfb5 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 078f8eff84c94dde9705ad1330922a11 (127.10.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7046cbd82d634976b53fe0cef5788f18" member_type: VOTER last_known_addr { host: "127.10.145.1" port: 35261 } } peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { replace: true } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } } peers { permanent_uuid: "e938d875af584f1f88a2d4b50c589268" member_type: VOTER last_known_addr { host: "127.10.145.8" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: NON_VOTER last_known_addr { host: "127.10.145.4" port: 45951 } attrs { promote: true } } }
I20240503 14:16:15.921440 377 raft_consensus.cc:1050] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11: attempting to promote NON_VOTER acb93936f2f14e628eecb05d7d1f4d91 to VOTER
I20240503 14:16:15.922696 377 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } }
I20240503 14:16:15.935206 30302 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:15.936125 28186 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:15.936544 750 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:15.936844 28446 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:15.937054 28319 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 LEARNER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:15.937994 485 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:15.938202 27794 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:15.938637 485 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:15.939098 27926 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:15.940208 580 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:15.940833 580 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:15.947086 485 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:15.960223 30165 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:16:15.961110 485 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:15.979521 766 ts_tablet_manager.cc:902] T bbb645fc3c5c422b9e3fabba8ce12873 P 01173ed91fd341e3a4a4599118167e52: Initiating tablet copy from peer 7046cbd82d634976b53fe0cef5788f18 (127.10.145.1:35261)
I20240503 14:16:15.983491 27926 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.982466 759 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.990471 27519 catalog_manager.cc:5484] T 9f00380931064f71bc78626af72488f3 P a242f99f54b4494893064cfe3030b4ad reported cstate change: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "078f8eff84c94dde9705ad1330922a11" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.992854 30302 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.992894 28446 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 9231e715a4e342a197d6c5456ba6b399 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.993911 28185 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:15.994757 30165 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 54dab61f1a204c0da4f56f98b3fd2690 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:16.003027 27794 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P 74318b6270b64efaa79867167fc08bbb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:16.014653 28055 consensus_queue.cc:237] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } }
I20240503 14:16:16.018167 28319 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:16.020171 28185 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P c91f285b8f4f4a27bcdd0ada303c1422 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:16.020578 750 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:16:16.020951 580 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:16:16.022557 28319 raft_consensus.cc:2932] T 9f00380931064f71bc78626af72488f3 P acb93936f2f14e628eecb05d7d1f4d91 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, acb93936f2f14e628eecb05d7d1f4d91 (127.10.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "74318b6270b64efaa79867167fc08bbb" member_type: VOTER last_known_addr { host: "127.10.145.2" port: 38859 } } peers { permanent_uuid: "a242f99f54b4494893064cfe3030b4ad" member_type: VOTER last_known_addr { host: "127.10.145.3" port: 36119 } } peers { permanent_uuid: "078f8eff84c94dde9705ad1330922a11" member_type: VOTER last_known_addr { host: "127.10.145.4" port: 45951 } } peers { permanent_uuid: "c91f285b8f4f4a27bcdd0ada303c1422" member_type: VOTER last_known_addr { host: "127.10.145.5" port: 35605 } } peers { permanent_uuid: "9231e715a4e342a197d6c5456ba6b399" member_type: VOTER last_known_addr { host: "127.10.145.7" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "54dab61f1a204c0da4f56f98b3fd2690" member_type: VOTER last_known_addr { host: "127.10.145.9" port: 39057 } attrs { promote: false } } peers { permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10" port: 40989 } attrs { promote: false } } peers { permanent_uuid: "acb93936f2f14e628eecb05d7d1f4d91" member_type: VOTER last_known_addr { host: "127.10.145.6" port: 36073 } attrs { promote: false } } }
I20240503 14:16:16.026094 30304 raft_consensus.cc:1261] T 9f00380931064f71bc78626af72488f3 P 01173ed91fd341e3a4a4599118167e52 [term 1 FOLLOWER]: Refusing update from remote peer 078f8eff84c94dde9705ad1330922a11: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 14:16:16.027326 766 tablet_copy_client.cc:308] T bbb645fc3c5c422b9e3fabba8ce12873 P 01173ed91fd341e3a4a4599118167e52: tablet copy: Beginning tablet copy session from remote peer at address 127.10.145.1:35261
I20240503 14:16:16.028128 27682 tablet_copy_service.cc:140] P 7046cbd82d634976b53fe0cef5788f18: Received BeginTabletCopySession request for tablet bbb645fc3c5c422b9e3fabba8ce12873 from peer 01173ed91fd341e3a4a4599118167e52 ({username='slave'} at 127.10.145.10:60537)
I20240503 14:16:16.028324 27682 tablet_copy_service.cc:161] P 7046cbd82d634976b53fe0cef5788f18: Beginning new tablet copy session on tablet bbb645fc3c5c422b9e3fabba8ce12873 from peer 01173ed91fd341e3a4a4599118167e52 at {username='slave'} at 127.10.145.10:60537: session id = 01173ed91fd341e3a4a4599118167e52-bbb645fc3c5c422b9e3fabba8ce12873
I20240503 14:16:16.028841 485 consensus_queue.cc:1035] T 9f00380931064f71bc78626af72488f3 P 078f8eff84c94dde9705ad1330922a11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01173ed91fd341e3a4a4599118167e52" member_type: VOTER last_known_addr { host: "127.10.145.10"